The first three lessons made your client retry less and stop more. All of that assumed something quietly: that repeating a call is harmless. For a lookup it is. For "charge this card", "send this email" or "insert this row" it is the difference between a reliable system and a support queue full of double charges.
Repeats are not rare edge cases. They come from four ordinary places:
- Retries after an ambiguous failure. Lesson one showed that a reset or a timeout after the request was sent leaves you not knowing whether the server acted.
- Queue redelivery. A worker processes a message, crashes before acknowledging it, and the queue hands the message to another worker.
- Restarts. Lesson three halted a job on credit exhaustion. When it resumes, it will meet work that was half done.
- Humans. Someone reruns yesterday's job because the dashboard looked wrong.
You cannot remove these. You can make them harmless.
At-least-once is the delivery you actually get
Every mainstream queue (SQS, Pub/Sub, RabbitMQ, Kafka consumer groups, BullMQ) delivers at least once. The reason is structural: the queue can only forget a message after the consumer acknowledges it, and the consumer can only acknowledge after it has done the work. A crash in the gap between "did the work" and "acknowledged" means the work is done and the queue still has the message.
"Exactly once" between two systems that fail independently is not something a queue can sell you. What you can build is effectively once: at-least-once delivery plus a consumer that recognises a repeat and does nothing the second time. That property is called idempotency, and it lives in your code, not in the queue.
An operation is idempotent when doing it twice leaves the world in the same state as doing it once. Some operations are idempotent by nature:
| Operation | Safe to replay? | What a replay costs |
|---|---|---|
A lookup (GET) on APIVerve | Yes | Credits again, and the answer may differ if the data moved |
An APIVerve POST transformation (HMAC, HTML to PDF, image conversion) | Yes | Credits again; nothing is created twice |
Setting a field to a value (status = 'shipped') | Yes | Nothing |
| Upsert keyed by a stable id | Yes | Nothing |
| Charging a card through a payment provider | Only with an idempotency key | A second charge |
| Sending an email or SMS | No, unless you dedupe | A second message to a customer |
| Incrementing a counter or appending a row | No | A wrong total |
The design job is to move every operation in the bottom half of that table into the top half.
APIVerve calls are on the safe side
APIVerve's POST endpoints are transformations: they take an input and return an output, and they create nothing on the platform that a second call would duplicate. Sending the same body twice gives you a second answer to the same question. That is why there is no idempotency key to send for them and none is needed. The HMAC Generator is a clean example: same message, same secret, same signature, every time.
curl -X POST "https://api.apiverve.com/v1/hmac" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"message":"order-1042:charge","secret":"whsec_demo"}'Run it twice: data.hmac is identical both times and nothing was created. A replay of a transformation costs credits, never correctness.
Replays of APIVerve calls still cost something: each successful call bills again, and a 10,000-item enrichment job that reruns from the start pays twice. Deduplicating them is a cost decision, not a correctness one. Two details follow:
- Some answers move over time. Validation, DNS-based checks, current time and live prices can differ between the first call and the replay. If downstream code must see one consistent answer, store the first result under the operation's key and serve the replay from the store instead of calling again.
- For batches, replay only what failed. Results come back in the order you sent them, so resend just the indices whose item failed with
code: 429(rate window) orcode: 402(credits), or that never got a response. Resending the whole batch re-bills every item that already succeeded.
Dedupe keys come from the operation
An idempotent consumer needs a key that is the same every time the same logical operation is attempted, and different for different operations. The most common mistake is generating it at the wrong moment:
// Wrong: a new key on every attempt makes every retry look like new work
await sendReceipt(order, { dedupeKey: crypto.randomUUID() });A random UUID is fine if it is created once, when the operation is first decided, and stored with the work (in the queue message, in the database row). It is useless if it is created inside the code that retries. The more robust approach is to derive the key from what the operation is:
import { createHash } from 'node:crypto';
// Stable JSON: same object, same string, whatever order the keys were added in.
function canonical(value) {
if (Array.isArray(value)) return `[${value.map(canonical).join(',')}]`;
if (value && typeof value === 'object') {
return `{${Object.keys(value).sort().map((k) => `${JSON.stringify(k)}:${canonical(value[k])}`).join(',')}}`;
}
return JSON.stringify(value);
}
/** Key for one logical operation. Never include timestamps, attempt numbers or random values. */
export function opKey(kind, identity) {
return createHash('sha256').update(`${kind}:${canonical(identity)}`).digest('hex');
}
opKey('receipt-email', { orderId: 1042 }); // same every time for this order
opKey('enrich-company', { domain: 'example.com' }); // same input, same key, one paid callWhat goes into identity is a business decision, and it is worth saying out loud for each operation. "Send a receipt for order 1042" is one operation no matter how many times the message arrives, so the key is the order id. "Send a reminder for order 1042" might legitimately happen once per day, so the key includes the date it is for (the business date, not the time of the attempt). If you cannot say what makes two attempts the same operation, you are not ready to dedupe it.
Hashing keeps keys a fixed length and keeps raw identifiers (emails, card fingerprints) out of logs and key tables. You can cross-check a digest against the Hash Generator while testing; sha256 of order-1042:charge should match what node:crypto produces:
curl "https://api.apiverve.com/v1/hashgenerator?text=order-1042%3Acharge&algorithms=sha256" \
-H "x-api-key: YOUR_API_KEY"data.hashes.sha256 equals createHash('sha256').update('order-1042:charge').digest('hex') in Node.
A worker sends a receipt email for each order message. It retries on timeouts and the queue redelivers on crashes. Which dedupe key prevents duplicate receipts?
The idempotent consumer
The key is only useful if checking it and claiming it happen atomically, otherwise two workers that receive the same redelivered message at the same moment both see "not done yet". A database unique constraint gives you that atomicity. This version uses Postgres and adds a lease, so that a worker which crashes mid-operation does not block the key forever:
CREATE TABLE operations (
op_key text PRIMARY KEY,
state text NOT NULL, -- 'running' | 'done'
lease_until timestamptz NOT NULL,
result jsonb
);import pg from 'pg';
import { opKey } from './op-key.mjs';
const db = new pg.Pool();
// Returns { owned: true } if this worker should do the work, or { owned: false, done, result }.
async function claim(key, leaseSeconds = 120) {
const { rows } = await db.query(
`INSERT INTO operations (op_key, state, lease_until)
VALUES ($1, 'running', now() + make_interval(secs => $2))
ON CONFLICT (op_key) DO UPDATE
SET state = 'running', lease_until = now() + make_interval(secs => $2)
WHERE operations.state <> 'done' AND operations.lease_until < now()
RETURNING op_key`,
[key, leaseSeconds],
);
if (rows.length) return { owned: true };
const existing = await db.query('SELECT state, result FROM operations WHERE op_key = $1', [key]);
return { owned: false, done: existing.rows[0]?.state === 'done', result: existing.rows[0]?.result };
}
async function complete(key, result) {
await db.query(`UPDATE operations SET state = 'done', result = $2 WHERE op_key = $1`, [key, result]);
}
export async function handleMessage(msg, { sendReceipt }) {
const key = opKey('receipt-email', { orderId: msg.orderId });
const c = await claim(key);
if (!c.owned) return c.done ? 'ack' : 'retry-later'; // done before, or someone else is on it
const result = await sendReceipt(msg.orderId, { idempotencyKey: key });
await complete(key, result);
return 'ack';
}Read the ON CONFLICT clause carefully, because it is the whole design. A first claim inserts the row. A repeat while another worker holds a live lease matches nothing and returns no row, so the second worker backs off. A repeat after the lease expired (the first worker died) takes the row over. A repeat after done never takes it over, and the caller acknowledges the message without doing anything.
There is still one gap: the worker can crash after sendReceipt succeeded and before complete ran. The lease expires, another worker claims the key, and the email goes again. A local database cannot close that gap on its own, because the side effect happens in someone else's system. Two things narrow it: pass the same key to the downstream provider when it supports idempotency keys (the next section), and keep the window between the side effect and complete as short as possible.
Providers that need an idempotency key
APIs that create things (payments, orders, messages, tickets) usually accept an idempotency key with the request. Stripe is the reference example: send the same Idempotency-Key header on a retried POST and Stripe returns the original result instead of creating a second charge. Its documentation keeps keys for at least 24 hours, and it rejects a reuse of the same key with different parameters, which catches bugs where two different operations accidentally share a key.
import Stripe from 'stripe';
import { opKey } from './op-key.mjs';
const stripe = new Stripe(process.env.STRIPE_SECRET_KEY);
export function chargeOrder(order) {
return stripe.paymentIntents.create(
{ amount: order.totalCents, currency: 'usd', customer: order.customerId, confirm: true,
payment_method: order.paymentMethodId },
{ idempotencyKey: opKey('charge', { orderId: order.id }) }, // same key on every retry
);
}The rules carry over to any provider that offers keys:
- Derive the key from the operation, as above, so a redelivered message and a manual rerun both reuse it.
- Reuse it on every retry of that operation, including retries after an ambiguous timeout. The ambiguous case is precisely what the key is for.
- Know the provider's retention window. A replay after the provider has forgotten the key is a new request. Your own
operationstable, which you control, is the defence for anything older. - Inbound webhooks need the mirror image. Providers deliver webhooks at least once too. Store each event's id and ignore ones you have processed, using the same claim-and-complete pattern.
For APIVerve you skip all of this: there is no key to send, a replay creates nothing, and the only thing to manage is the credit cost of the repeat.
A call to an APIVerve POST endpoint (HTML to PDF) times out after the request was sent. What is the correct retry behaviour?
The outbox pattern
Idempotent consumers solve repeats on the receiving side. The sending side has its own classic bug, the dual write: your code updates its database and then publishes a message, and a crash between the two leaves them disagreeing. Order saved, no "order placed" message, no receipt, no enrichment. Publishing first is no better: message sent, order never saved.
The outbox pattern removes the second write from the critical path. The message is written into an outbox table in the same database transaction as the business change. Either both commit or neither does. A separate relay process then publishes outbox rows and marks them sent.
export async function placeOrder(db, order) {
const client = await db.connect();
try {
await client.query('BEGIN');
const { rows } = await client.query(
'INSERT INTO orders (customer_id, total_cents) VALUES ($1, $2) RETURNING id', [order.customerId, order.totalCents]);
await client.query(
`INSERT INTO outbox (topic, payload) VALUES ('order-placed', $1)`, [{ orderId: rows[0].id }]);
await client.query('COMMIT');
return rows[0].id;
} catch (err) {
await client.query('ROLLBACK');
throw err;
} finally {
client.release();
}
}
// Relay: run on a timer or a loop. SKIP LOCKED lets several relays share the table safely.
export async function relayOnce(db, publish) {
const client = await db.connect();
try {
await client.query('BEGIN');
const { rows } = await client.query(
`SELECT id, topic, payload FROM outbox WHERE published_at IS NULL
ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED`);
for (const row of rows) await publish(row.topic, row.payload);
if (rows.length) {
await client.query('UPDATE outbox SET published_at = now() WHERE id = ANY($1)', [rows.map((r) => r.id)]);
}
await client.query('COMMIT');
} catch (err) {
await client.query('ROLLBACK');
throw err;
} finally {
client.release();
}
}Notice that the relay is itself at-least-once: if it crashes after publish and before COMMIT, the same rows are published again on the next run. That is fine, and it is the point. The outbox guarantees the message is never lost; the idempotent consumer guarantees a duplicate is never acted on twice. Together they give effectively-once processing end to end, with each half doing only what it can actually guarantee.
What you have now, and what comes next
This lesson closes the course. You can now classify every failure your client sees into one action, bound retries with deadlines, budgets and breakers, pace a fleet under one shared limit while forecasting credits, and make every operation safe to replay: lookups and APIVerve transformations by nature, side effects through operation-derived keys, an idempotent consumer and the outbox pattern.
Resilience keeps an integration alive. The next course is about keeping it affordable as volume grows: controlling what you call, how often, and what it costs each month. Continue with Scale and cost control.