Production API Caching: Cache Keys, Single-Flight, Redis

The Fundamentals course ended with a small TTL cache: a Map, a sorted key, the pending Promise stored so concurrent callers share one call, failures evicted. That is the right shape for one process. It starts to leak money the moment the integration grows up:

Ten server instances each keep their own Map, so a cold deploy pays ten times for the same country record.
Callers build the same question in different ways (USD and usd, a stray utm_source, parameters in another order) and every variant is a separate entry, a separate call, and a separate charge.
An entry expires at 09:00:00 and every request in that second waits on the refresh, so your p99 latency has a sawtooth that lines up with your TTLs.
A lookup that finds nothing is never cached, so a bad value typed into a form is sent upstream on every keystroke.

This lesson fixes each of those. The pieces are independent; take the ones your traffic needs.

Decide the TTL from how fast the data moves

A TTL is a claim about the source: "this answer will still be true for N seconds". Get it from the data, not from a round number. Five classes cover almost every APIVerve endpoint:

Class	Example	What changes it	Starting TTL
Pure function of the input	Number to Words, Hash Generator, Query String Builder	Nothing. Same input, same output, forever	No expiry, bounded by size
Reference data	Country Lookup, currency symbols, ISO codes	A treaty, a new currency. Years	7 to 30 days
Slowly moving records	DNS, WHOIS, company records, job listings	The owner edits them. Hours	1 to 24 hours
Live values	Exchange rates, weather, market data	The world, continuously	Seconds to minutes, tied to the source's own update time
Per-request generation or verdicts	Random data, UUIDs, Email Validator	Every call is meant to differ, or describes this instant	Do not cache

Two refinements matter at scale.

Read the source's own clock when it has one. Exchange Rate returns lastUpdated. If the rate was refreshed four minutes ago and the source updates every five, a five-minute TTL from your fetch time can serve a value that is nine minutes old. Expire relative to the source's timestamp when the response carries one, and fall back to your fetch time when it does not.

Add jitter. If a deploy warms 50,000 keys in the same minute with the same 24-hour TTL, they all expire in the same minute tomorrow, and you get a stampede of misses at the same hour every day. Spread expiry with a random 10 percent:

ttl.mjs

export const ttlWithJitter = (ms, spread = 0.1) => Math.round(ms * (1 - spread / 2 + Math.random() * spread));

Build one canonical key for one question

A cache key must be equal exactly when the answers would be equal. Two failure modes pull in opposite directions: a key that is too specific splits one question into many entries (you pay for each), and a key that is too general merges different questions (you serve one user another user's answer). The second is a correctness bug, so every normalisation step must be one you are sure the API treats as equivalent.

Start with the query string. ?a=1&b=2 and ?b=2&a=1 are the same request, but string comparison says otherwise. A canonical query string sorts the keys, encodes every value the same way, skips parameters that carry no value, and has one fixed rule for arrays:

query.mjs

export function buildQuery(params = {}) {
  return Object.keys(params)
    .filter((k) => params[k] !== null && params[k] !== undefined)
    .sort()
    .map((k) => {
      const v = params[k];
      const encoded = Array.isArray(v)
        ? v.map((item) => encodeURIComponent(String(item))).join(',')
        : encodeURIComponent(String(v));
      return `${encodeURIComponent(k)}=${encoded}`;
    })
    .join('&');
}

buildQuery({ b: 2, a: 1 });                          // "a=1&b=2"
buildQuery({ a: 1, b: 2, c: undefined, d: null });   // "a=1&b=2"
buildQuery({ text: 'rock & roll', algorithms: ['sha256', 'md5'] });
// "algorithms=sha256,md5&text=rock%20%26%20roll"

Each rule has a reason:

Sorted keys make parameter order irrelevant. The default sort() compares UTF-16 code units, which is deterministic on every machine, and that is all a key needs.
encodeURIComponent on every value means rock & roll can never be read as two parameters, and the same value always produces the same bytes. Pick one encoder and use it everywhere: URLSearchParams writes a space as + while encodeURIComponent writes %20, so mixing them splits the cache.
null and undefined are skipped, so an optional filter the caller left unset produces the same key as a caller who never mentioned it. An empty string is kept: '' is a value, and on some APIs it means something.
Arrays are encoded per item and joined with a literal comma. Many APIVerve parameters take comma-separated lists, such as Hash Generator's algorithms=sha256,md5. Encoding each item first means a comma inside one item becomes %2C, so ['a,b'] and ['a', 'b'] stay distinct keys. Item order is kept, because for some parameters order means something.

APIVerve's own Query String Builder is useful for checking your encoding against a second implementation. Note what it does with order:

Try it liveQuery String Builder1 credit

POSThttps://api.apiverve.com/v1/querystringbuilder

curl -X POST "https://api.apiverve.com/v1/querystringbuilder" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"params":{"to":"EUR","from":"USD"}}'

params

data.queryString keeps the order you gave: to=EUR&from=USD. A builder that preserves order is right for a URL and wrong for a cache key, which is why buildQuery sorts first.

Normalise per API, then drop what the API ignores

Sorting fixes order. It does not fix usd versus USD, or a tracking parameter that rode in from a URL. Put per-API normalisers in front of buildQuery:

cache-key.mjs

import { buildQuery } from './query.mjs';

// Declared parameters per API, from each reference page. Anything else is dropped.
const DECLARED = {
  countrylookup: ['country', 'majorcities'],
  exchangerate: ['currency1', 'currency2', 'date'],
};

const NORMALISE = {
  exchangerate: (p) => ({ ...p, currency1: p.currency1?.trim().toUpperCase(), currency2: p.currency2?.trim().toUpperCase() }),
};

export function cacheKey(api, params) {
  const allowed = DECLARED[api];
  const kept = allowed ? Object.fromEntries(Object.entries(params).filter(([k]) => allowed.includes(k))) : params;
  const normalised = NORMALISE[api] ? NORMALISE[api](kept) : kept;
  return `${api}?${buildQuery(normalised)}`;
}

The allow-list is not tidiness. APIVerve silently drops parameters an endpoint does not declare: no 400, the call simply runs without them. So country=JP&ref=newsletter returns exactly what country=JP returns, and keeping ref in the key pays twice for one answer. The same silence cuts the other way. A misspelt parameter (majorCities for majorcities) is dropped too, so the API answers a different question than you meant, and the key, which includes the typo, faithfully caches the wrong answer. Building keys from the declared list catches that in review, because the typo vanishes from the key.

Upper-casing currency codes is safe because ISO 4217 codes are defined in capitals. Lower-casing an email address is not obviously safe: the domain is case-insensitive, the local part technically is not. When you are unsure, do not normalise; an extra entry costs credits, a merged entry costs correctness.

Keys can get long (a 2,000-character text parameter is a 2,000-character key). Hash them for storage, and keep the API id as a readable prefix so you can still scan and expire by API:

Try it liveHash Generator1 credit

GEThttps://api.apiverve.com/v1/hashgenerator

1
2

curl "https://api.apiverve.com/v1/hashgenerator?text=countrylookup%3Fcountry%3DJP&algorithms=sha256" \
  -H "x-api-key: YOUR_API_KEY"

textalgorithms

data.hashes.sha256 is a fixed 64-character key for any input length. Store it as countrylookup:<hash>.

In Node the same digest is one line, with no call needed in production: createHash('sha256').update(key).digest('hex') from node:crypto.

Check yourself

Your cache key is built as api + '?' + new URLSearchParams(params). Hit rate on Exchange Rate is far lower than expected. Which change fixes the most likely cause without risking wrong answers?

One upstream call per key, whatever the burst

The Fundamentals cache stored the pending Promise in the cache itself. That works until you add a second layer or a stale value: the TTL store and the "who is already fetching this" question are different concerns. Keep them apart. An in-flight map holds only calls that are running right now:

single-flight.mjs

const inFlight = new Map();

export function singleFlight(key, fn) {
  const running = inFlight.get(key);
  if (running) return running;
  const p = Promise.resolve()
    .then(fn)
    .finally(() => inFlight.delete(key));
  inFlight.set(key, p);
  return p;
}

Every caller that arrives while the fetch runs gets the same Promise. When it settles, success or failure, the entry is removed, so a failure is shared by the callers who were already waiting but never cached for anyone who arrives later. The Promise.resolve().then(fn) wrapper means a function that throws synchronously still produces a rejected Promise instead of escaping before the map is updated.

This is per process. Across instances, a cold deploy still sends one call per instance per key. Usually that is fine: ten instances paying once each is a bounded cost. When it is not (an expensive endpoint, hundreds of instances) add a short lock in the shared cache, shown below.

Serve stale while you refresh

A hard TTL makes the request that finds an expired entry pay the full upstream latency. Stale-while-revalidate gives each entry two deadlines:

fresh until: serve from cache, no call.
stale until: serve the cached value immediately and start one background refresh.
after that: the value is too old to show; wait for a fresh fetch.

It also gives you stale-if-error for free: if the background refresh fails, the old value stays in place and keeps being served until its stale deadline, so a short upstream incident does not become your incident.

swr-cache.mjs

import { singleFlight } from './single-flight.mjs';
import { ttlWithJitter } from './ttl.mjs';

const store = new Map(); // key -> { value, freshUntil, staleUntil }

export async function swr(key, fetcher, { freshMs, staleMs }) {
  const now = Date.now();
  const hit = store.get(key);

  const refresh = () => singleFlight(key, async () => {
    const value = await fetcher();
    const t = Date.now();
    store.set(key, { value, freshUntil: t + ttlWithJitter(freshMs), staleUntil: t + freshMs + staleMs });
    return value;
  });

  if (hit && now < hit.freshUntil) return hit.value;
  if (hit && now < hit.staleUntil) {
    refresh().catch(() => {}); // keep serving stale if the refresh fails
    return hit.value;
  }
  return refresh();
}

Choose staleMs per data class, the same way as the TTL. For country records a week of staleness is invisible. For exchange rates, serving a 20-minute-old rate while a refresh runs may be fine on a dashboard and wrong at checkout. Stale-while-revalidate is a product decision about how old an answer may be, not just a latency trick.

Cache "not found", briefly

Some APIVerve endpoints answer an input that matches nothing with a 404 and an error envelope. Country Lookup does this:

Try it liveCountry Lookup2 credits

GEThttps://api.apiverve.com/v1/countrylookup

1
2

curl "https://api.apiverve.com/v1/countrylookup?country=Atlantis" \
  -H "x-api-key: YOUR_API_KEY"

country

HTTP 404 with error 'country not found' and data null. The call failed, so it was not billed, but it still used a request.

A failed call is not billed, so negative caching here is not mostly about credits. It is about the other two things a repeated miss costs:

Rate-limit slots. An autocomplete that looks up every keystroke, or a job that keeps meeting the same bad row, spends your per-minute window on answers you already have.
The repeated-failure throttle. When the same call keeps failing the same way, APIVerve throttles that exact call and replays the original error. A negative cache means you never get close.

On endpoints that answer "nothing found" with a successful, empty result, an empty answer is billed, and negative caching saves credits as well.

Rules for doing it safely:

Negative-cache only the "nothing found for this input" answer, on endpoints where you have seen that is what a 404 means. A 404 because the API id is wrong is a bug, and it fails for every input, so you will see it in testing.
Give it a short TTL, minutes to an hour. Absence is less stable than presence: the record may be created later.
Never cache 400 (fix the input instead), 401, 403, 429 or 5xx. Those describe your key, your pace or the platform at that moment, not the input.

negative.mjs

import { swr } from './swr-cache.mjs';

const NOT_FOUND = Symbol('not-found');

export async function lookup(key, call) {
  return swr(key, async () => {
    try { return await call(); }
    catch (err) {
      if (err.status === 404) return NOT_FOUND;   // cached like a value
      throw err;                                  // everything else stays uncached
    }
  }, { freshMs: 10 * 60_000, staleMs: 0 });
}

In practice give negative entries their own shorter freshMs than positive ones; the sketch uses one value to stay short. (swr is the function from the previous section.)

An in-process cache is the fastest layer and the least shared one. Redis (or any shared key-value store) is one network hop away and shared by every instance, so a value fetched once is reused everywhere and survives a deploy. Use both: memory as L1 for the hottest keys, Redis as L2.

shared-cache.mjs

import { createClient } from 'redis';
import { createHash } from 'node:crypto';

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

const storageKey = (key) => `${key.split('?')[0]}:${createHash('sha256').update(key).digest('hex')}`;

export async function sharedGet(key, fetcher, ttlSeconds) {
  const k = storageKey(key);
  const cached = await redis.get(k);
  if (cached !== null) return JSON.parse(cached);

  // Short lock so only one instance refreshes a cold key.
  const gotLock = await redis.set(`${k}:lock`, '1', { NX: true, PX: 10_000 });
  if (!gotLock) {
    await new Promise((r) => setTimeout(r, 200));
    const retry = await redis.get(k);
    if (retry !== null) return JSON.parse(retry);
  }
  const value = await fetcher();
  await redis.set(k, JSON.stringify(value), { EX: ttlSeconds });
  return value;
}

The lock is advisory: if the instance holding it dies, PX expires it, and a waiter that still finds nothing after its pause fetches anyway. That is the correct failure mode, because a lock that can block a request forever is worse than an occasional duplicate call.

Three operational rules for a shared cache:

Store the envelope's data, not the HTTP response. Headers such as x-api-remaining-credits describe the account at fetch time and are meaningless when replayed later.
Premium fields are cached as they arrived. On APIVerve a field your plan does not include comes back as null, not absent. If two keys on different plans share one cache, the lower plan's null can be served to the higher plan. Partition the cache by plan, or only share across keys on the same plan.
Fail open. If Redis is down, call the API. A cache outage should cost credits, not availability. Wrap every Redis call so an error falls through to fetcher().

Check yourself

Ten app instances share a Redis cache. A swr wrapper runs in each instance with a per-process single-flight map. A cold key gets 500 concurrent requests spread evenly across the instances. Without the Redis lock, how many upstream calls should you expect at most?

Python: the same canonical key

If part of your stack is Python, the key function must produce byte-identical keys or the two halves of your system will not share a single entry:

query.py

from urllib.parse import quote

SAFE = "-_.!~*'()"   # what encodeURIComponent leaves alone


def build_query(params):
    parts = []
    for k in sorted(k for k, v in params.items() if v is not None):
        v = params[k]
        if isinstance(v, (list, tuple)):
            encoded = ",".join(quote(str(item), safe=SAFE) for item in v)
        else:
            encoded = quote(str(v), safe=SAFE)
        parts.append(quote(k, safe=SAFE) + "=" + encoded)
    return "&".join(parts)


print(build_query({"b": 2, "a": 1}))   # a=1&b=2

quote with that safe set matches encodeURIComponent, which leaves the same nine characters unescaped. Watch two traps: Python's str(True) is "True" where JavaScript's is "true", and sorted() on strings orders by code point where JavaScript orders by UTF-16 unit, which only differs for characters outside the Basic Multilingual Plane. Convert booleans explicitly, and add a shared test vector to both test suites so a drift fails a build rather than a cache.

Caching removes calls you have already paid for. The next lesson handles the calls you do need when there are thousands of them: Batch requests.

Finished the lesson?Mark it complete to track your progress toward the certification.

Cache API responses in production

Decide the TTL from how fast the data moves

Build one canonical key for one question

Normalise per API, then drop what the API ignores

One upstream call per key, whatever the burst

Serve stale while you refresh

Cache "not found", briefly

Python: the same canonical key

Run every example on your own key. Free plan, no card.

Scaling up?

Decide the TTL from how fast the data moves

Build one canonical key for one question

Normalise per API, then drop what the API ignores

One upstream call per key, whatever the burst

Serve stale while you refresh

Cache "not found", briefly

Share the cache across instances

Python: the same canonical key

Run every example on your own key. Free plan, no card.

Scaling up?