APIVerveAcademy
Start for free
APV-300Scale and Cost Control
Lesson 3 of 4

Bulk jobs and pagination

Run a large API enrichment job that checkpoints and resumes, paces itself to the rate limit, and merges paginated results without duplicates.

  • 45 min
  • Advanced

A 100,000-row enrichment job is not a bigger version of a 500-row one. At 500 rows you run a loop, and if it dies you run it again. At 100,000 rows the job runs for hours, it will be interrupted (a deploy, a laptop lid, an out-of-credits refusal at row 61,000), and running it again from the top pays for the first 61,000 rows twice. The work in this lesson is making that impossible.

Two kinds of volume show up in practice, and they need different tools:

  • Many inputs, one answer each. A CSV of 100,000 emails, domains or IPs. You control the list; the job is chunking, pacing and resuming.
  • One input, many answers. A listing that comes back a page at a time, like every open job at a company. The API controls the list; the job is walking pages and merging them correctly.

Do the arithmetic before the first call

Two numbers decide whether a bulk job is feasible: credits and time. Credits are lesson 4. Time is set by the rate limit, and batching does not change it, because a batch commits one slot of your per-minute window for every item it runs.

At a window of L requests per minute, N lookups take at least N / L minutes. With 70,000 unique inputs:

Window (requests/minute)Minimum run time
60about 19.5 hours
180about 6.5 hours
No per-minute limitbounded by latency and your own concurrency

Read the window your key actually has from x-rate-limit-limit on any response (server-side; browsers cannot read it), or from rateLimit in the free GET /v1/analytics summary, which reports a string like 60 requests/minute or unlimited.

Two decisions follow from that table:

  • Deduplicate first. It cuts time and credits in the same proportion. Lesson 2's planBatches does it.
  • Give the job its own sub-key. A sub-key has its own rate-limit window, so a backfill that saturates its window does not starve the interactive traffic on your main key. It still spends the account's shared credits; it fences off rate, not money. On Mega you can also cap a sub-key's per-minute rate below the plan ceiling.

Checkpoints keyed by input, not by position

The tempting checkpoint is a row number: "processed up to 61,000". It breaks as soon as the input changes between runs (a re-export reorders rows, someone removes a duplicate) and it cannot express "rows 60,801 to 61,000 were sent but the response never arrived".

Checkpoint by the canonical key of each input instead, in an append-only file of results. On start, the file tells you which keys are done; everything else is still to do. A crash can lose at most the batch that was in flight.

checkpoint.mjs
import { createReadStream } from 'node:fs';
import { appendFile } from 'node:fs/promises';
import { createInterface } from 'node:readline';

export async function loadDone(path) {
  const done = new Map();                       // key -> envelope
  try {
    const lines = createInterface({ input: createReadStream(path), crlfDelay: Infinity });
    for await (const line of lines) {
      if (!line.trim()) continue;
      try {
        const { key, result } = JSON.parse(line);
        done.set(key, result);
      } catch { /* a torn last line from a crash: ignore it, the key will be redone */ }
    }
  } catch (err) {
    if (err.code !== 'ENOENT') throw err;       // first run: no file yet
  }
  return done;
}

export async function record(path, entries) {
  // One appendFile per batch: a crash loses at most this batch, never earlier ones.
  await appendFile(path, entries.map((e) => JSON.stringify(e)).join('\n') + '\n');
}

JSON Lines suits this well: appending never rewrites earlier lines, and a crash in the middle of a write damages only the last line, which loadDone skips. Store errors that are the input's own answer (an invalid email, nothing found) in the same file, so a resume does not pay rate limit to hear the same verdict again. Do not record 429 or 402 items: they were never answered, and the next run must send them.

The job

This ties lesson 1's buildQuery, lesson 2's batch sender and the checkpoint together:

enrich.mjs
import { buildQuery } from './query.mjs';
import { runBatches } from './run-batches.mjs';
import { loadDone, record } from './checkpoint.mjs';

export async function enrich(api, rows, toParams, { out = `${api}.results.jsonl`, chunk = 200 } = {}) {
  const done = await loadDone(out);

  // Unique, not-yet-done inputs, in first-seen order.
  const todo = new Map();
  for (const row of rows) {
    const params = toParams(row);
    const key = buildQuery(params);
    if (!done.has(key) && !todo.has(key)) todo.set(key, params);
  }
  console.log(`${done.size} already done, ${todo.size} to run`);

  const keys = [...todo.keys()];
  for (let i = 0; i < keys.length; i += chunk) {
    const slice = keys.slice(i, i + chunk);
    const { results, pending } = await runBatches(api, slice.map((k) => todo.get(k)), { size: chunk });
    const entries = [...results].map(([j, result]) => ({ key: slice[j], result }));
    await record(out, entries);
    for (const e of entries) done.set(e.key, e.result);
    if (pending.length) throw new Error(`${pending.length} items still rate-limited; rerun to resume`);
  }

  // Join back to the input, in input order: one answer per row, duplicates included.
  return rows.map((row) => done.get(buildQuery(toParams(row))) ?? null);
}

Three properties make this safe to kill and rerun at any point:

  • Idempotent by key. Rerunning skips every key in the results file. A row that appears 400 times in the input is looked up once and answered 400 times in the join.
  • Bounded loss. Results are recorded per batch, before the next batch starts. The worst case after a crash is one batch sent again.
  • Output order is input order. Work happens in whatever order is efficient; the final map over rows restores the original order. Never rely on completion order for anything a person will read.

When runBatches throws on 402, the completed results from that round are attached to the error as done. Catch it in enrich and record them before rethrowing if you want to keep the last partial batch too; otherwise those few items are simply redone on the next run.

Concurrency: fewer workers than you think

Running several batches in parallel looks like the obvious speed-up. Against one key it mostly is not: every batch draws from the same per-minute window, so two parallel batches reach the limit twice as fast and then both wait. Concurrency helps with latency, not with rate. It pays when the endpoint is slow, so that one batch's wait overlaps another's work, or when the key has no per-minute limit.

When you do run in parallel, cap it with a small pool rather than an unbounded Promise.all:

pool.mjs
export async function pool(tasks, limit) {
  const results = new Array(tasks.length);
  let next = 0;
  async function worker() {
    while (next < tasks.length) {
      const i = next++;
      results[i] = await tasks[i]();
    }
  }
  await Promise.all(Array.from({ length: Math.min(limit, tasks.length) }, worker));
  return results;
}

Start at two workers for batches and measure. If rateLimited climbs, you have more concurrency than window; adding workers will only add 429s and lengthen the edge's cooldown on the key.

Check yourself

A resumable job checkpoints the last processed row number. Between two runs, someone re-exports the input CSV sorted by a different column. What is the most likely result of resuming?

Run every example on your own key. Free plan, no card.

Get a free API key→All courses

Scaling up?

Volume pricing, custom SLAs, and dedicated support for high-traffic teams.

Contact sales