By now a single call in your system is well behaved. It has a deadline, it retries only what a second attempt can fix, and it stops dead on a 429 with zero credits left. That is necessary, and at system level it is not enough. A checkout page that calls five APIs has five dependencies that will each, at some point this year, be slow or unavailable for longer than any retry loop should wait. The question stops being "how do I make this call succeed?" and becomes "what does this feature do when the call cannot succeed?"
That question has a design answer, and it should be written down before the outage, not improvised during it. This lesson covers the four pieces of that answer: a failure posture chosen per feature, a fallback chain that walks from best answer to honest failure, a circuit breaker that decides when to stop trying, and the isolation (bulkheads and timeout budgets) that keeps one sick dependency from taking the healthy ones down with it.
Decide the posture per feature, not per API
Every dependency failure ends in one of two places. Fail open means the feature proceeds as if the check passed or the enrichment was optional. Fail closed means the feature stops, or routes the case somewhere slower and safer, until the dependency answers.
Neither is correct in general. The choice comes from comparing two costs: what a wrong "yes" costs you, and what a wrong "no" costs you. Work through it for each feature that depends on an external call:
| Feature | Dependency | Wrong yes costs | Wrong no costs | Posture |
|---|---|---|---|---|
| Weather widget on a dashboard | Weather forecast | Nothing; there is no yes | A blank card | Fail open: hide or show stale |
| Signup email check | Email validation | A junk account you can clean later | A real customer turned away | Fail open, flag for re-check |
| Signup VPN or datacenter signal | VPN detection | Some fraud gets one step further | Friction for honest users | Fail to a softer path (extra verification) |
| Payout to a new payee | Sanctions screening | A possible legal violation | A delayed payout | Fail closed: hold as pending |
| Card-not-present fraud score | Several signals combined | A chargeback | A lost sale | Depends on order value; decide with the business |
Two things fall out of the table. First, the same API can sit under features with different postures. An IP lookup that only personalises a landing page fails open; the same lookup feeding a fraud rule may not. Posture belongs to the feature, so it lives in the feature's code, not in a shared client wrapper. Second, "fail closed" rarely means "show an error". It usually means "put this case in a state that a later process or a human will resolve": a payout marked pending, an order held for review. Fail closed is a queue, not a wall.
A marketplace pays sellers weekly. The payout job calls a sanctions screen for any payee whose details changed. The screening dependency is timing out. What posture should the payout feature take?
Build the fallback chain
Once a feature can fail open, the next question is how gracefully. A fallback chain is an ordered list of ways to produce an answer, from best to least good, and the feature takes the first one that works:
- Primary provider, live. The normal path.
- Secondary provider, live. A different source for the same fact. It may cost more, differ in fields or be less precise; the code that consumes it must tolerate that.
- Stale cache. The last good answer, served past its normal freshness window, up to a hard maximum age that you choose per data type. Exchange rates from an hour ago are fine for a price preview and wrong for a settlement.
- Reduced feature. Show the parts that do not need the dependency. A product page without shipping estimates is still a product page.
- Honest unavailable. Say the thing is unavailable, and when it is likely to come back. Never invent a default that looks like a real answer: a weather card reading 0 degrees is worse than one reading "unavailable".
Label degraded answers in the response your own service returns, for example { source: "stale", asOf: "..." }, so the UI can say "as of 10:40" and so your metrics can count how often you degraded. A system that silently serves stale data for a week is not resilient; it is broken in a way nobody can see.
What should trigger a fallback
Only availability failures move you down the chain: a timeout, a network error, a 5xx, a rate-limit 429, or an open circuit. An answer never should. When an API says your input is bad (400) or that nothing was found (404 on some lookups), asking a second provider usually gets the same answer, at extra cost, and hides a bug in your input. Try both calls below and compare the status of the second:
curl "https://api.apiverve.com/v1/weatherforecast?city=Denver" \
-H "x-api-key: YOUR_API_KEY"A normal answer. In a widget you would cache data.tempF, data.condition and the time you fetched them.
curl "https://api.apiverve.com/v1/weatherforecast?city=NotARealCityzzz" \
-H "x-api-key: YOUR_API_KEY"This returns 404 with error 'No matching location found.' That is an answer about the input, not an outage. Do not fail over on it.
One 429 deserves its own line. A rate-limit 429 on APIVerve is a reason to wait, and a stale cache can cover the wait. A 429 with x-api-remaining-credits at 0 means the account's allowance is spent: every APIVerve call on that account will fail the same way until renewal, so failing over to another APIVerve API is pointless, while a different provider or the cache remains valid. Make the chain aware of that difference.
Choosing a provider from health
Keep the selection logic pure: give it the candidates in priority order and the current health of each, and let it return a choice. Health is computed elsewhere (by circuit breakers, below), which keeps both pieces testable.
// providers: [{ id: 'primary', priority: 1, kind: 'live' }, { id: 'cache', priority: 9, kind: 'stale' }]
// health: { primary: false, secondary: true, cache: true } (missing means unknown: treat as unhealthy)
export function chooseProvider(providers, health) {
const ordered = [...providers].sort((a, b) => a.priority - b.priority); // never mutate the caller's array
const live = ordered.find((p) => p.kind !== 'stale' && health[p.id] === true);
if (live) return live;
const stale = ordered.find((p) => p.kind === 'stale' && health[p.id] === true);
return stale ?? null; // null: reduced feature or honest "unavailable"
}
// Usage
const providers = [
{ id: 'secondary', priority: 2, kind: 'live' },
{ id: 'primary', priority: 1, kind: 'live' },
{ id: 'cache', priority: 9, kind: 'stale' },
];
chooseProvider(providers, { primary: false, secondary: true, cache: true }); // secondary
chooseProvider(providers, { primary: false, secondary: false, cache: true }); // cache
chooseProvider(providers, { primary: false, secondary: false, cache: false }); // nullNotice three deliberate choices. The function copies before sorting, because sorting in place would reorder a configuration array shared by every request. A provider with no health entry counts as unhealthy, because "unknown" is not "fine". And a stale source is only picked when no live source is healthy, whatever priority number someone gave it, so a misconfigured priority cannot make the cache win over a working provider. For the cache, "healthy" means "has an entry younger than the maximum stale age"; the caller computes that and passes it in.
Circuit breakers as a pure state machine
APV-300 used a breaker to stop hammering a failing API. At system level the breaker does a second job: its state is the health signal that the fallback chain reads. That makes it worth writing as a pure function, with no timers and no I/O, so that you can unit test every transition and store the state wherever your instances can share it (memory for one process, a key-value store for a fleet).
The three states:
- Closed. Calls flow. Consecutive failures are counted; a success resets the count. When the count reaches the threshold, the breaker opens.
- Open. Calls are refused without touching the network. After a cooldown, the breaker lets one trial through.
- Half-open. One trial call is in flight. Success closes the breaker; failure reopens it and restarts the cooldown.
// state: { status: 'closed' | 'open' | 'half-open', failures: number, openedAt: number | null }
// event: 'request' (asking permission), 'success' or 'failure' (reporting an outcome)
export const initialBreaker = { status: 'closed', failures: 0, openedAt: null };
export function breaker(state, event, nowMs, { threshold = 5, cooldownMs = 30_000 } = {}) {
const s = { ...state };
// An open breaker whose cooldown has elapsed becomes half-open, whatever the event.
if (s.status === 'open' && nowMs - s.openedAt >= cooldownMs) s.status = 'half-open';
if (event === 'success') {
if (s.status === 'open') return s; // a late result from before it opened changes nothing
return { status: 'closed', failures: 0, openedAt: null };
}
if (event === 'failure') {
if (s.status === 'half-open') return { status: 'open', failures: s.failures, openedAt: nowMs };
if (s.status === 'open') return s;
const failures = s.failures + 1;
return failures >= threshold
? { status: 'open', failures, openedAt: nowMs }
: { status: 'closed', failures, openedAt: null };
}
return s; // 'request' only applies the cooldown check
}
export const allows = (s) => s.status !== 'open';A caller asks first, then reports:
import { breaker, allows, initialBreaker } from './breaker.mjs';
const states = new Map(); // dependency id -> breaker state
export async function guarded(dep, fn, opts) {
let s = breaker(states.get(dep) ?? initialBreaker, 'request', Date.now(), opts);
states.set(dep, s);
if (!allows(s)) throw Object.assign(new Error(`${dep} circuit open`), { circuitOpen: true });
try {
const out = await fn();
states.set(dep, breaker(states.get(dep), 'success', Date.now(), opts));
return out;
} catch (err) {
if (err.countsAsFailure !== false) states.set(dep, breaker(states.get(dep), 'failure', Date.now(), opts));
throw err;
}
}
export const health = (deps) =>
Object.fromEntries(deps.map((d) => [d, allows(breaker(states.get(d) ?? initialBreaker, 'request', Date.now()))]));Two details decide whether the breaker helps or hurts. First, only availability failures count. Mark answers (400, 404, an out-of-credits 429) with countsAsFailure: false, or a burst of bad user input will open the breaker and take the feature down for everyone. Second, the simple version above lets every concurrent caller through during half-open. With heavy traffic, add a flag so only one trial is in flight, or the "trial" becomes a stampede against a dependency that is just recovering.
Bulkheads: one slow API cannot sink the ship
A dependency that fails fast is easy to live with. One that becomes slow is the dangerous one, because slow calls hold resources: sockets, memory for pending Promises, worker threads, database connections held open while waiting, and, in a serverless function, the function's own concurrency. If a 20-second stall in one optional enrichment can consume all of those, every feature in the process fails, including the ones that never call it.
A bulkhead gives each dependency its own fixed allowance of concurrent calls. When the allowance is full, new calls are rejected immediately (or wait in a short, bounded queue), and the rejection flows into the fallback chain like any other availability failure.
export function bulkhead(maxConcurrent, maxQueue = 0) {
let active = 0;
const waiting = [];
const next = () => {
if (active < maxConcurrent && waiting.length) { active++; waiting.shift()(); }
};
return async function run(fn) {
if (active >= maxConcurrent) {
if (waiting.length >= maxQueue) throw Object.assign(new Error('bulkhead full'), { bulkheadFull: true });
await new Promise((resolve) => waiting.push(resolve));
} else {
active++;
}
try { return await fn(); } finally { active--; next(); }
};
}
// One compartment per dependency, sized to what each deserves.
export const pools = {
weather: bulkhead(4), // optional enrichment: small, no queue
vpn: bulkhead(10, 20), // on the signup path: larger, short queue
};Size compartments from what each dependency is worth, not from what it asks for. An optional widget gets a small pool and no queue, so under stress it degrades first and gives its share to something that matters. Keep the total of all compartments below the process's real limits, and remember the account-level limit too: concurrency beyond your plan's per-minute rate only converts into 429s, which the next lesson turns into arithmetic.
Timeouts inside a request budget
Per-call timeouts from APV-300 answer "how long may this call take?". A request that makes several calls needs a second answer: "how long may the whole request take?". If your signup endpoint promises an answer within 2 seconds, and it calls an email check with a 10-second timeout, the per-call timeout is a lie.
Treat the request's deadline as a budget that each step draws from. Each dependency's effective timeout is the smaller of its own normal timeout and what remains of the budget, minus a reserve for the work after it (writing to your database, rendering the response).
export function budget(totalMs, reserveMs = 150) {
const deadline = Date.now() + totalMs;
return {
remaining: () => Math.max(0, deadline - Date.now() - reserveMs),
timeout(ownMs) {
const ms = Math.min(ownMs, this.remaining());
if (ms <= 0) throw Object.assign(new Error('request budget spent'), { budgetSpent: true });
return AbortSignal.timeout(ms);
},
};
}
// Independent checks run in parallel and share the budget.
export async function signupSignals(b, verve, { email, ip }) {
const [mail, net] = await Promise.allSettled([
verve('emailvalidator', { email }, { signal: b.timeout(1500) }),
verve('vpndetector', { ip }, { signal: b.timeout(1500) }),
]);
return {
email: mail.status === 'fulfilled' ? mail.value : { degraded: true },
network: net.status === 'fulfilled' ? net.value : { degraded: true },
};
}Promise.allSettled matters here: with Promise.all, one rejected check would discard the other's good result. Each signal is then judged by its own feature's posture: a degraded email check fails open, a degraded network signal routes the signup to extra verification.
curl "https://api.apiverve.com/v1/vpndetector?ip=8.8.8.8" \
-H "x-api-key: YOUR_API_KEY"is_datacenter is true for a public DNS resolver. In a signup flow this is a signal with a posture of its own, not a verdict.
Sequential calls make budgets tighter: if the second call depends on the first, the first must leave room for the second. When the arithmetic does not fit, that is a design finding, not a tuning problem. Move the slower step out of the request (run it after signup and act on the result asynchronously) instead of raising the budget until users notice.
A request has a 2,000 ms budget with a 150 ms reserve. 1,400 ms have already been spent on an earlier step. The next dependency normally uses a 1,500 ms timeout. What timeout should the next call get?
Putting the pieces together
The pieces compose in a fixed order for each dependency call: the budget sets the timeout, the bulkhead admits or rejects, the breaker admits or rejects and records the outcome, and the fallback chain reads breaker health to pick a source and catches whatever the first three throw. The feature's posture decides what "nothing worked" means.
Test this the way it will fail. In a staging environment, wrap the client so a flag can make one dependency slow (sleep past its timeout), erroring (503) or answering wrongly (400), and check three things: healthy features stay healthy, degraded responses are labelled, and the breaker opens on the first two cases but not the third.
Everything here assumed one request at a time from one user. Real systems serve many tenants who share one account's limits and credits, and the failure that bites first is often your own traffic. Next: Queues and fair rate limits for many tenants.