APV-300 taught you to stay under a rate limit from one process: pace calls, honour retry-after, batch where it pays. This lesson starts where that stops. Your product has many customers. Some import 40,000 contacts at 09:00 on a Monday; most do nothing for days. All of them share one APIVerve account, which means one credit allowance and, unless you split it, one per-minute window. The architecture questions are now: where does a burst wait, how many workers should drain it, who goes first, and what does the system say when it is genuinely full?
Level the load with a queue
A synchronous design calls the API inside the user's request. That is right for small, interactive work where the user is waiting for the answer. It is wrong for anything a single user can make large, because the request rate your users generate is unrelated to the request rate your plan allows. The import of 40,000 contacts either runs at the API's pace (and the HTTP request times out) or runs at the user's pace (and hits 429s within seconds).
Queue-based load levelling separates the two rates. The request handler writes one job per unit of work to a durable queue and returns at once, typically 202 Accepted with a job id the client can poll or subscribe to. A pool of workers takes jobs from the queue and calls the API at a pace you control. The burst still arrives; it just waits in the queue instead of in your users' browsers or in the provider's rate limiter.
Three properties make a queue worth having rather than merely present:
- Durability. Jobs survive a deploy or a crashed worker. A job is acknowledged only after its result is stored, so a worker that dies mid-job leaves it to be picked up again.
- Visibility. You can see depth and, more usefully, the age of the oldest job. Depth tells you how much work is waiting; age tells you whether anyone is being failed.
- Bounded size. A queue with no limit is not load levelling; it is a delay you have not measured yet. The section on backpressure covers what to do at the bound.
Because a job can run twice (a worker can crash after calling the API and before acknowledging), make each job safe to repeat. APIVerve's endpoints are lookups and transformations, so a repeat costs credits but changes nothing on the provider's side. Your own side effects are the risk: write results with an upsert keyed by the job id, not an insert.
Size the worker pool with Little's law
How many workers should drain the queue? Not "as many as possible": workers beyond what your rate limit allows only produce 429s. Not "one": a single worker spends most of its time waiting on network latency. The answer comes from Little's law, which holds for any stable system:
L = λ × W
in-flight requests = arrival rate × time each request spends in the systemTurn it around for a worker pool. If you want to sustain a throughput of λ requests per second and each call takes W seconds, you need L calls in flight, which means L workers (each worker holds one call at a time).
Work it through with real numbers. At the time of writing, APIVerve's published per-minute limits are 5 on Free, 60 on Starter and 180 on Pro, with no per-minute ceiling on Mega (check rate limits for the current table; the effective limit can also be lowered per key). Take a Pro account:
- Target throughput: 180 per minute = 3 requests per second.
- Typical latency of the call: 0.6 seconds. L = 3 × 0.6 = 1.8, so 2 workers keep the limit busy.
- Latency on a bad day: 2.5 seconds. L = 3 × 2.5 = 7.5, so 8 workers are needed to hold the same throughput.
That gap is the point. Size the pool for the latency you will see under stress (a high percentile, not the median), and let a pacer be the thing that enforces the rate. On a good day, most of the 8 workers wait on the pacer; on a bad day, they are all busy and the rate still holds. Without a pacer, 8 workers on a good day would try to send 13 requests per second into a 3 per second limit.
The pacer should refill continuously rather than in one-minute chunks. APIVerve's limiter works the same way, so a steady stream spread across the minute never trips a ceiling that the same number of calls fired in one second usually does. A token bucket with a small burst allowance is the standard shape:
// A token bucket shared by every worker in the process.
export function pacer(perMinute, burst = Math.max(1, Math.round(perMinute / 20))) {
const ratePerMs = perMinute / 60_000;
let tokens = burst;
let last = Date.now();
let pausedUntil = 0;
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
return {
async take() {
for (;;) {
const now = Date.now();
if (now < pausedUntil) { await sleep(pausedUntil - now); continue; }
tokens = Math.min(burst, tokens + (now - last) * ratePerMs);
last = now;
if (tokens >= 1) { tokens -= 1; return; }
await sleep(Math.ceil((1 - tokens) / ratePerMs));
}
},
// A rate-limit 429 pauses everyone, not just the worker that saw it.
pause(seconds) { pausedUntil = Math.max(pausedUntil, Date.now() + seconds * 1000); tokens = 0; },
};
}Two operational rules go with it. Set the pacer a little under the plan limit (say 90%), because interactive traffic on the same key also counts. And when any worker gets a rate-limit 429, pause the whole pool for retry-after seconds. The edge holds a rate-limited key in a short cooldown of around 50 seconds, and a pool that keeps sending during it extends the lockout for everyone.
Little's law also gives you drain time. A backlog of 40,000 jobs at 3 per second takes about 3.7 hours, before any new work arrives. That is a number to put in front of the product team before the first big customer imports, not after.
Credits are often the tighter limit
Rate is per minute; credits are per month, and they vary by API. Email Validator costs 20 credits a call. Running it flat out on Pro is 180 × 20 = 3,600 credits a minute, which would spend Pro's 1,000,000 monthly credits in under five hours. For expensive APIs, the per-minute ceiling is not what bounds you; the monthly allowance is, and your scheduler must meter credits as well as calls.
curl "https://api.apiverve.com/v1/emailvalidator?email=jane%40mailinator.com" \
-H "x-api-key: YOUR_API_KEY"isDisposable is true. Note the x-api-remaining-credits header drop: this API costs more than most, which is what your per-tenant ledger must count.
A worker pool calls an API at a paced 2 requests per second. Under load, each call takes 3 seconds. How many workers do you need to hold 2 requests per second, and what stops them exceeding it on a fast day?
Many tenants, one limit
Put every tenant's jobs in one first-in, first-out queue and the first big import blocks everyone. A tenant who queued 40,000 jobs at 09:00 makes a tenant who queued 3 jobs at 09:01 wait 3.7 hours. Nothing failed, and the system is still unfair. This is the noisy neighbour problem, and the fix has four parts.
One queue per tenant, served in rotation. The scheduler visits tenants round-robin and takes one job from each that has work. A tenant with 3 jobs finishes in seconds regardless of who else is busy. Weighted rotation (take two jobs from a higher tier per turn) expresses paid priority without letting it starve anyone.
Per-tenant concurrency caps. No tenant may hold more than a fixed share of the worker pool at once. Rotation decides order; the cap decides how much of the pool a single tenant can occupy while its jobs are slow.
Per-tenant credit quotas. Your account has one allowance; your customers should each have a slice. Meter it in credits, not calls, using each API's credit cost, because one tenant running sanctions screens at 20 credits a call consumes the pool ten times faster than another tenant running exchange rates at 2. When a tenant's slice is spent, their jobs wait or fail with a clear "quota reached" while everyone else carries on.
Separate windows for separate workloads. APIVerve sub-keys bill to the same account, so they share the credit allowance, but each has its own rate-limit window. Running bulk imports on one sub-key and interactive traffic on another means a saturated import pool cannot consume the window your users' live requests depend on. A sub-key can also be revoked on its own, which matters when one workload misbehaves. See sub-keys for which restrictions each plan allows.
Here is a scheduler that combines rotation, caps and a credit ledger. It is in-memory for clarity; in production the queues and the ledger live in your database or queue service, and the same logic runs against them.
export function fairScheduler({ perTenantMax = 2, quotas = {}, cost = {} }) {
const queues = new Map(); // tenant -> [job]
const active = new Map(); // tenant -> in-flight count
const spent = new Map(); // tenant -> credits used this period
let ring = []; // rotation order
let cursor = 0;
return {
enqueue(tenant, job) {
if (!queues.has(tenant)) { queues.set(tenant, []); ring.push(tenant); }
queues.get(tenant).push(job);
},
// Returns the next job that is allowed to run now, or null.
next() {
for (let i = 0; i < ring.length; i++) {
const tenant = ring[(cursor + i) % ring.length];
const q = queues.get(tenant);
if (!q.length || (active.get(tenant) ?? 0) >= perTenantMax) continue;
const credits = cost[q[0].api] ?? 1;
const quota = quotas[tenant] ?? Infinity;
if ((spent.get(tenant) ?? 0) + credits > quota) continue; // this tenant waits, others proceed
cursor = (cursor + i + 1) % ring.length;
active.set(tenant, (active.get(tenant) ?? 0) + 1);
spent.set(tenant, (spent.get(tenant) ?? 0) + credits); // reserve now; refund on failure
return { tenant, job: q.shift(), credits };
}
return null;
},
done(tenant, { billed = true, credits = 0 } = {}) {
active.set(tenant, active.get(tenant) - 1);
if (!billed) spent.set(tenant, spent.get(tenant) - credits); // failed calls are not billed
},
depth: () => Object.fromEntries([...queues].map(([t, q]) => [t, q.length])),
};
}Each worker loops: await pacer.take(), const slot = scheduler.next(), call the API, then scheduler.done(slot.tenant, ...). Reserving credits at dispatch time, not on completion, stops a tenant from overshooting its quota with many calls in flight. Refunding on a failed call keeps the ledger honest: APIVerve bills successful calls, and a 400 or a 429 should not count against a customer.
The cost map comes from each API's credit cost on its marketplace page or in its definition. Keep it in configuration, not scattered through code, so a pricing change is a one-line edit.
Backpressure: saying "not now" on purpose
Every queue fills eventually. Backpressure is the set of decisions you make before it does, so the system degrades in a way you chose instead of the way memory and timeouts choose for you.
Bound every queue, per tenant and overall. When a tenant's queue is full, reject new jobs at the front door with your own 429 and a Retry-After. When the overall queue is full, do the same for the lowest priority work first.
Admit by priority. Not all jobs are equal. A live signup check outranks a nightly re-verification of old contacts. Give the scheduler two or three priority classes and let the lowest class shed first. Shedding a nightly job costs a delay; shedding a signup check costs a customer.
Watch age, not length. A queue of 10,000 cheap jobs that drains in a minute is fine. A queue of 50 jobs where the oldest is two hours old is an incident. Alert on oldest-job age per priority class, compared with the promise you made for that class.
Respond to the provider's signals across the pool. A rate-limit 429 pauses the whole pacer for retry-after. A 429 with x-api-remaining-credits at 0 stops all dispatch for that account and pages a human: no amount of waiting fixes it before renewal. A breaker opening (lesson one) stops dispatch for that dependency only, while jobs for other APIs continue.
Batch inside the worker, carefully. When a tenant's queue holds many jobs for the same endpoint, a worker can send up to 200 of them as one POST /v1/<api>/batch. Deduplicate the inputs first, since duplicates bill each time. Remember what a batch does to your pacer: it commits one rate-limit slot per item, not per HTTP call, and items beyond the minute's capacity come back as per-item 429s with retryAfter inside an HTTP 200. Items beyond your remaining credits come back as per-item 402s. Your worker must walk results and re-queue exactly those items, with the tenant's credit reservations refunded for the ones that did not bill.
Tenant A has queued 30,000 sanctions screens (20 credits each). Tenant B has queued 50 exchange-rate lookups (2 credits each). Both share one account. Which design keeps B fast without letting A exceed its share of the monthly allowance?
What you have now
Bursts land in durable, bounded queues. A pool sized by Little's law, throttled by a shared pacer, drains them at the rate your plan allows. Tenants are served in rotation, capped in concurrency and metered in credits, and when the system is full it says so early and sheds the least important work first.
All of that moves data: emails, names, IP addresses and phone numbers flow through queues, logs, caches and third-party APIs. The next lesson is about deciding what should flow at all, and making sure what does can be accounted for. Next: Personal data handling and compliance for APIs.