The Fundamentals course gave you a retry loop: four tries, exponential backoff with jitter, honour retry-after, never retry credit exhaustion. Inside one script that is correct. Inside a fleet it can still take you down, in three ways that each have a known fix.
- Each attempt has a timeout, but the call as a whole does not. Four tries of 10 seconds plus 8 seconds of backoff is 48 seconds, while the user who triggered it gave up after five. The fix is an overall deadline.
- Every caller retries independently. When an upstream starts failing, every client multiplies its load by its retry count at the exact moment the upstream can least take it. The fix is a retry budget.
- Nobody remembers that the last thousand calls failed. Each new request starts optimistic, waits out its full timeout, and fails. The fix is a circuit breaker.
This lesson builds all three as small pieces, then composes them with the error classes from lesson one.
Retry amplification
Retries multiply across layers. Suppose a browser calls your API with 3 attempts, your API calls a worker service with 3 attempts, and the worker calls APIVerve with 4. If APIVerve's upstream for one endpoint is failing, one user click can produce 3 x 3 x 4 = 36 calls to it. Every one of those either costs latency or, if it succeeds partway through, credits.
Two rules keep this under control:
- Retry at one layer only. Pick the layer closest to the failing dependency (the code that calls APIVerve) and make every layer above it fail fast. A service that calls another of your services should usually not retry at all; it should pass the error and its retryability upward.
- Make the remaining layer's retries conditional on the fleet's state, not just on this request's history. That is the budget and the breaker.
An overall deadline
A deadline is a single timestamp computed when the operation starts. Every attempt's timeout is the smaller of its own limit and the time left, and no backoff sleep is allowed to end after the deadline.
export function makeDeadline(totalMs, now = Date.now()) {
const at = now + totalMs;
return {
at,
remaining: (t = Date.now()) => Math.max(0, at - t),
// Per-attempt timeout: never longer than the time that is left.
attemptTimeout: (perAttemptMs, t = Date.now()) => Math.min(perAttemptMs, Math.max(0, at - t)),
// Is it worth sleeping `waitMs` and then trying again?
allows: (waitMs, minAttemptMs = 500, t = Date.now()) => t + waitMs + minAttemptMs <= at,
};
}The minAttemptMs term in allows() is easy to miss. Sleeping until 50 ms before the deadline and then starting an attempt with a 50 ms timeout is a wasted call: it spends a rate-limit slot and cannot finish. If there is not enough time left for a realistic attempt after the wait, give up now and return the last error.
When the deadline comes from upstream (a web request with its own timeout, a queue message with a visibility window), derive yours from it with some margin rather than inventing an independent number. A worker whose message becomes visible again after 30 seconds should finish, successfully or not, well inside 30 seconds, or a second worker will start the same job.
Full jitter as a pure function
The delay formula from Fundamentals is worth turning into a function you can test. Injecting the random number instead of calling Math.random() inside makes it deterministic in tests:
/**
* Full-jitter exponential backoff.
* @param {number} attempt zero-based: 0 for the wait after the first failure
* @param {number} baseMs ceiling for attempt 0
* @param {number} capMs maximum ceiling for any attempt
* @param {number} rand a number in [0, 1), normally Math.random()
* @returns {number} delay in milliseconds, in [0, min(capMs, baseMs * 2 ** attempt))
*/
export function backoffDelay(attempt, baseMs, capMs, rand) {
const ceiling = Math.min(capMs, baseMs * 2 ** attempt);
return rand * ceiling;
}Working through a few values makes the shape concrete. With baseMs = 100 and capMs = 1000:
attempt | baseMs * 2 ** attempt | ceiling | rand = 0.5 gives |
|---|---|---|---|
| 0 | 100 | 100 | 50 |
| 1 | 200 | 200 | 100 |
| 3 | 800 | 800 | 400 |
| 4 | 1600 | 1000 | 500 |
| 40 | about 1.1e14 | 1000 | 500 |
The cap is what keeps large attempt numbers sane. In JavaScript 2 ** 1024 is Infinity, and Math.min(capMs, Infinity) is still capMs, so the function never returns NaN or overflows. Clamp the attempt anyway if you log the uncapped value.
Why full jitter rather than "exponential plus a little random"? With a small jitter on top of a fixed exponential schedule, a thousand clients that failed together still return in tight waves at 1 s, 2 s, 4 s. Full jitter spreads each wave across the entire interval from zero to the ceiling, so the load the upstream sees after an incident is roughly flat. The cost is that some individual retries come back almost immediately, which is fine: the budget below stops that from becoming a flood.
The same function in Python, for completeness:
def backoff_delay(attempt: int, base_ms: float, cap_ms: float, rand: float) -> float:
return rand * min(cap_ms, base_ms * 2 ** attempt)Python integers do not overflow, so a huge attempt produces a huge integer before min reduces it. Harmless, but clamp attempt if you call it in a loop with no upper bound.
backoffDelay(attempt, baseMs, capMs, rand) returns rand * min(capMs, baseMs * 2 ** attempt). What is backoffDelay(5, 200, 3000, 0.25)?
Honouring Retry-After in both formats
When a server says how long to wait, its number beats your formula. HTTP defines two formats for Retry-After: a non-negative integer number of seconds, or an HTTP-date such as Wed, 22 Oct 2025 12:00:00 GMT. APIVerve sends seconds. On a rate-limit 429 from the origin it is the time until the oldest request in your one-minute window ages out; when the edge answers for a key it is holding in a cooldown, it is the time left on that cooldown. Batch items that were rate limited carry the same information in a retryAfter field, also in seconds. Other providers you call alongside APIVerve may send the date form, so a general parser handles both.
The trap is Date.parse. It is extremely lenient, and it will happily read a seconds value as a date:
Date.parse('120'); // -58380372564000: the year 120
Date.parse('1.5'); // a date in January 2001
Date.parse('-5'); // a date in 2001So the order of the checks is not a detail. Test for the numeric form first, and only fall back to date parsing for values that look like a date.
/**
* @param {string|null|undefined} value raw Retry-After header
* @param {number} nowMs current time in ms (Date.now())
* @returns {number|null} milliseconds to wait, or null when absent or unparseable
*/
export function parseRetryAfter(value, nowMs) {
if (value === null || value === undefined) return null;
const text = String(value).trim();
if (text === '') return null;
if (/^\d+$/.test(text)) return Number(text) * 1000; // delay-seconds
if (/[a-z]/i.test(text)) { // looks like an HTTP-date
const at = Date.parse(text);
if (!Number.isNaN(at)) return Math.max(0, at - nowMs); // a past date means "now"
}
return null; // '-5', '1.5', 'soon', ...
}A date in the past returns 0, not a negative number: the server's moment has already come, so retry immediately (with your own jitter). Anything that is neither a whole number of seconds nor a parseable date returns null, and the caller falls back to backoffDelay. Two further rules belong in the caller, not the parser: cap the hint (a misconfigured server that says "retry in 86400 seconds" should not park a worker for a day; if the hint exceeds your deadline, fail now) and add a little jitter on top, so clients told the same number do not all return in the same millisecond.
The HTTP-date form is always GMT. The Unix Timestamp Converter returns exactly this shape in its rfc_2822 field, which is a convenient way to produce test fixtures:
curl "https://api.apiverve.com/v1/unixtimestamp?timestamp=1761134400" \
-H "x-api-key: YOUR_API_KEY"data.rfc_2822 is Wed, 22 Oct 2025 12:00:00 GMT, the same format an HTTP-date Retry-After uses. parseRetryAfter(that, 1761134370000) should return 30000.
The Python standard library already has a parser for that format:
import re
from email.utils import parsedate_to_datetime
def parse_retry_after(value, now_ms):
if value is None:
return None
text = str(value).strip()
if re.fullmatch(r"\d+", text):
return int(text) * 1000
try:
at = parsedate_to_datetime(text)
except (TypeError, ValueError):
return None
if at is None or at.tzinfo is None:
return None
return max(0, int(at.timestamp() * 1000) - now_ms)A third-party API returns Retry-After: 120. Your parser calls Date.parse(value) first and only uses the numeric form if that fails. What happens?
Retry budgets
A per-request limit ("at most 4 attempts") does not bound total load: a thousand failing requests still produce four thousand calls. A retry budget bounds retries as a fraction of real traffic across the whole process. The common design, used by gRPC and several service meshes, is a token bucket that first requests fill and retries drain:
export class RetryBudget {
/**
* @param {object} o
* @param {number} o.ratio tokens earned per first attempt (0.1 = retries up to ~10% of traffic)
* @param {number} o.maxTokens cap, so a quiet hour does not bank thousands of retries
* @param {number} o.minPerSecond floor, so low-traffic clients can still retry occasionally
*/
constructor({ ratio = 0.1, maxTokens = 20, minPerSecond = 0.5 } = {}) {
Object.assign(this, { ratio, maxTokens, minPerSecond });
this.tokens = maxTokens;
this.last = Date.now();
}
#refill(now) {
this.tokens = Math.min(this.maxTokens, this.tokens + ((now - this.last) / 1000) * this.minPerSecond);
this.last = now;
}
onFirstAttempt(now = Date.now()) {
this.#refill(now);
this.tokens = Math.min(this.maxTokens, this.tokens + this.ratio);
}
tryRetry(now = Date.now()) {
this.#refill(now);
if (this.tokens < 1) return false;
this.tokens -= 1;
return true;
}
}In normal operation almost nothing fails, the bucket sits full, and every retryable failure gets its retry. During an outage the bucket drains within seconds and the process falls back to "one attempt per request, plus a trickle", which is exactly the load an overloaded upstream needs. A retry denied by the budget is not a new error type: the caller gets the original error, as if it were the last attempt.
Keep one budget per dependency (per API id is a good default) so that a failing endpoint cannot spend the retries another endpoint needs. Rate-limit 429s are the one retry that should not draw from this budget: they are not a sign of upstream failure, and the next lesson paces them with a limiter instead.
Circuit breakers
A budget limits extra load. A circuit breaker removes even the first attempt once a dependency is clearly down, so callers fail in microseconds instead of waiting out a timeout, and the upstream gets room to recover.
It has three states:
| State | Behaviour | Moves to |
|---|---|---|
| Closed | Calls go through. Consecutive failures are counted; a success resets the count. | Open, when the count reaches a threshold inside the window |
| Open | Calls fail immediately with a CircuitOpenError. Nothing is sent. | Half-open, after a cool-off period |
| Half-open | Exactly one probe call is allowed through; everything else still fails fast. | Closed if the probe succeeds, open again if it fails |
export class CircuitOpenError extends Error {
get retryable() { return false; }
get action() { return 'page'; }
}
export class CircuitBreaker {
constructor({ threshold = 5, windowMs = 30_000, openMs = 20_000, now = () => Date.now() } = {}) {
Object.assign(this, { threshold, windowMs, openMs, now });
this.state = 'closed';
this.failures = []; // consecutive failure timestamps, pruned to the window
this.openedAt = 0;
this.probeInFlight = false;
}
canRequest() {
const t = this.now();
if (this.state === 'open' && t - this.openedAt >= this.openMs) {
this.state = 'half-open';
this.probeInFlight = false;
}
if (this.state === 'closed') return true;
if (this.state === 'half-open' && !this.probeInFlight) {
this.probeInFlight = true; // this caller is the probe
return true;
}
return false;
}
onSuccess() { // any answer that says the upstream is healthy
this.state = 'closed';
this.failures = [];
this.probeInFlight = false;
}
onFailure() { // only failures that say the upstream is unhealthy
const t = this.now();
if (this.state === 'half-open') {
this.state = 'open';
this.openedAt = t;
this.probeInFlight = false;
return;
}
this.failures = this.failures.filter((f) => t - f < this.windowMs);
this.failures.push(t);
if (this.failures.length >= this.threshold) {
this.state = 'open';
this.openedAt = t;
}
}
}The most important decision is what counts as a failure. Count 5xx, timeouts, transport errors and non-JSON responses: they say the dependency is unhealthy. Do not count 400, 403 or 404; the upstream answered correctly, and a burst of bad input should never stop good input from getting through. Do not count rate-limit 429s either. They describe your pace, not the upstream's health, and opening a breaker on them would hide a pacing bug behind a fake outage. Credit exhaustion is its own hard stop from lesson one.
This breaker counts consecutive failures, which is easy to reason about and right for moderate traffic. At high volume, where a healthy endpoint still fails now and then, production libraries such as opossum for Node trip on a failure percentage over a rolling window with a minimum number of calls, so one unlucky failure in ten thousand never opens the circuit and a 40% failure rate always does. The states and transitions are the same; only the trip condition changes.
Scope breakers per API id. Different APIVerve endpoints depend on different sources: a 503 saying one endpoint's upstream source is unavailable tells you nothing about the others. One breaker for "APIVerve" would turn one source's bad afternoon into an outage of every feature you have.
Putting it together
Here is the composed call. It uses classifyResponse and classifyThrown from lesson one, and every piece of this lesson.
import { classifyResponse, classifyThrown } from './classify.mjs';
import { RateLimitError, TransportError, UpstreamError, BadResponseError } from './errors.mjs';
import { makeDeadline } from './deadline.mjs';
import { backoffDelay } from './backoff.mjs';
import { parseRetryAfter } from './retry-after.mjs';
import { RetryBudget } from './retry-budget.mjs';
import { CircuitBreaker, CircuitOpenError } from './breaker.mjs';
const breakers = new Map();
const budgets = new Map();
const per = (map, id, make) => map.get(id) ?? map.set(id, make()).get(id);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
const unhealthy = (e) => e instanceof UpstreamError || e instanceof TransportError || e instanceof BadResponseError;
export async function resilientCall(api, params, {
totalMs = 20_000, attemptMs = 8_000, maxAttempts = 4, baseMs = 250, capMs = 5_000, safeToRepeat = true,
} = {}) {
const deadline = makeDeadline(totalMs);
const breaker = per(breakers, api, () => new CircuitBreaker());
const budget = per(budgets, api, () => new RetryBudget());
const url = `https://api.apiverve.com/v1/${api}?${new URLSearchParams(params)}`;
budget.onFirstAttempt();
for (let attempt = 0; ; attempt++) {
if (!breaker.canRequest()) throw new CircuitOpenError(`${api}: circuit open, failing fast`);
let err;
try {
const res = await fetch(url, {
headers: { 'x-api-key': process.env.APIVERVE_API_KEY },
signal: AbortSignal.timeout(deadline.attemptTimeout(attemptMs)),
});
const body = await res.json().catch(() => null);
if (res.ok && body?.status === 'ok') { breaker.onSuccess(); return body.data; }
err = classifyResponse(res, body, api);
} catch (thrown) {
err = classifyThrown(thrown, api);
}
if (unhealthy(err)) breaker.onFailure();
else if (breaker.state === 'half-open') breaker.onSuccess(); // it answered: healthy enough
if (!err.retryable || attempt + 1 >= maxAttempts) throw err;
if (err instanceof TransportError && err.sent !== 'no' && !safeToRepeat) throw err;
const isRateLimit = err instanceof RateLimitError;
if (!isRateLimit && !budget.tryRetry()) throw err;
const hint = isRateLimit ? parseRetryAfter(err.retryAfter, Date.now()) : null;
const wait = hint !== null ? hint + Math.random() * 250 : backoffDelay(attempt, baseMs, capMs, Math.random());
if (!deadline.allows(wait)) throw err;
await sleep(wait);
}
}Read the order of the checks once more. The breaker is consulted before every attempt, not just the first, so a call that is mid-retry when the circuit opens stops immediately. Non-retryable errors leave on the first failure. An ambiguous transport failure on an unsafe operation is never repeated. Rate limits are paced by the server's hint and do not spend the budget; everything else must earn its retry from the budget. And no sleep is allowed to run the call past its deadline.
An endpoint starts returning 503 for every call. Your breaker counts 5xx, has threshold 5 in 30 seconds and openMs 20 seconds. What do callers experience over the next minute?
What you have now
Your client now knows when to stop: a deadline bounds each operation, full jitter spreads the retries it does make, Retry-After is parsed safely in both formats, a budget caps retries as a share of traffic, and a per-API breaker fails fast when an upstream is down. What it does not yet do is stay under the per-minute limit in the first place, or notice that credits are running low before they run out. Next: Rate limits and credits at scale.