A multi-API system is a machine for copying data. A signup form's email address goes to a validation API, into a queue, into a worker's logs, into an error tracker when something throws, into a cache keyed by the address, into your database, and into an analytics event. Each copy is somewhere the data can leak from, somewhere a deletion request has to reach, and something a privacy review will ask you about. Architecture decides how many copies exist. This lesson is about making that number small on purpose and making every remaining copy accountable.
Minimise at the call boundary
Data minimisation is the principle that you collect and pass on only what a purpose needs. At the API boundary it becomes a concrete rule: each outbound call sends exactly the fields that API requires, and nothing else. An email validator needs an email address. It does not need the person's name, their IP address or their account id, even though all of those are sitting in the same object when you make the call.
The failure mode is convenience. Someone writes verve('emailvalidator', user) with the whole user record, and because APIVerve silently drops parameters an endpoint does not declare, nothing breaks. The extra fields still crossed the wire, still sat in a query string, and may have been logged by any proxy, HTTP client or tracing library on the way out. Make the allowed fields explicit, per API, in one place:
// What each API may receive. Anything else is a bug, not a convenience.
const CONTRACT = {
emailvalidator: ['email'],
phonenumbervalidator: ['number', 'country'],
vpndetector: ['ip'],
sanctionscreening: ['name', 'lists'],
};
export function project(api, record) {
const allowed = CONTRACT[api];
if (!allowed) throw new Error(`No outbound contract for ${api}`);
const out = {};
for (const k of allowed) if (record[k] != null) out[k] = record[k];
return out;
}
// verve('emailvalidator', project('emailvalidator', user)) sends { email } and nothing moreThe contract doubles as documentation: it is the "what data goes to which processor" answer that a privacy review asks for, and it lives in code where it cannot drift from reality.
Minimise in the other direction too. Responses echo inputs. Email Validator returns the address in data.email and data.normalizedEmail; Sanctions Screening returns the screened name in data.query. If you cache or store whole responses, you have stored the personal data again. Project responses down to the fields you act on (isValid, isDisposable, verdict) before they go anywhere persistent.
GET endpoints put parameters in the URL, and URLs are the most logged string in computing. Configure your HTTP client, APM agent and reverse proxy to record the path without the query string for outbound calls, and treat any tool that captures full outbound URLs by default as a PII store until proven otherwise.
Keep personal data out of logs, traces and caches
Logs are where minimisation usually fails, because they are written for the worst day: the more you log, the easier the incident. The answer is not to log less but to log redacted structure. A log line that says a signup failed for user u_83f2 with an email at example.com and a 429 is almost as useful as one with the full address, and it is not a leak.
Redact a record without mutating it
Write one redaction function and route every structured log, error report and trace attribute through it. It must handle nesting and arrays, it must not mutate the object it is given (that object is often still in use by the request that is logging it), and it must match field names reliably:
const MASK = '[REDACTED]';
export function redact(record, fields) {
const targets = new Set(fields.map((f) => f.toLowerCase()));
const walk = (value) => {
if (Array.isArray(value)) return value.map(walk);
if (value !== null && typeof value === 'object' && !(value instanceof Date)) {
const out = {};
for (const [k, v] of Object.entries(value)) {
out[k] = targets.has(k.toLowerCase()) ? (v == null ? v : MASK) : walk(v);
}
return out;
}
return value;
};
return walk(record);
}
const event = {
userId: 'u_83f2',
contact: { Email: 'jane.roe@example.com', phone: '+14155552671' },
attempts: [{ ip: '203.0.113.42', status: 429 }],
card: null,
};
const safe = redact(event, ['email', 'phone', 'ip', 'card']);
// { userId: 'u_83f2', contact: { Email: '[REDACTED]', phone: '[REDACTED]' },
// attempts: [ { ip: '[REDACTED]', status: 429 } ], card: null }
// event is unchanged.Three decisions are worth noticing. Matching is case-insensitive, because Email and email arrive from different code paths. A null stays null, so the redacted log still tells you the field was absent rather than claiming it had a value. And the function builds new objects all the way down instead of copying the top level and editing in place, which is the classic way a "redacted" log ends up redacting the live request too.
The same function in Python, for log pipelines that run there:
MASK = "[REDACTED]"
def redact(record, fields):
targets = {f.lower() for f in fields}
def walk(value):
if isinstance(value, list):
return [walk(v) for v in value]
if isinstance(value, dict):
return {
k: (v if v is None else MASK) if k.lower() in targets else walk(v)
for k, v in value.items()
}
return value
return walk(record)When full masking is too much
Sometimes support needs to find "the failed signup from that Gmail address" or count failures per user without seeing who they are. Two techniques keep that ability without storing the value:
- Partial masks keep a non-identifying part: the email domain, the last four digits of a phone number. Useful for humans reading logs.
- Keyed hashes replace a value with
HMAC-SHA256(secret, normalisedValue). The same input always produces the same token, so you can join and count, but nobody without the secret can reverse it. A plain unkeyed hash is not enough for emails or phone numbers: the space of plausible inputs is small enough to hash them all and look the answer up.
import { createHmac } from 'node:crypto';
const SECRET = process.env.PII_HASH_SECRET; // rotate deliberately: old tokens stop matching
export const token = (value) =>
createHmac('sha256', SECRET).update(String(value).trim().toLowerCase()).digest('hex').slice(0, 24);
export const maskEmail = (email) => {
const at = String(email).lastIndexOf('@');
return at > 0 ? `***@${email.slice(at + 1)}` : '***';
};Use the keyed hash for cache keys too. A cache keyed by raw email addresses is a list of your users' email addresses with a TTL. Keyed by token(email) and holding only the projected response fields, it holds nothing personal, and a deletion request is handled by recomputing the token and deleting that key.
Free text is the hard case
Structured fields can be redacted by name. Free text cannot: a support ticket body or a third-party error message can contain anything. For that, pattern-based masking is the practical tool, and Data Masking (datamasking) does it as an API:
curl -X POST "https://api.apiverve.com/v1/datamasking" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"text":"checkout failed for jane.roe@example.com card 4111 1111 1111 1111 from 203.0.113.42, callback 555-123-4567, cc ops@example.org"}'masked replaces each match with a placeholder such as [EMAIL] or [CREDIT_CARD]; detected counts what was found (email: 2 here).
Be clear-eyed about the trade. Calling a masking API sends the unmasked text to another processor in order to clean it, and it costs 10 credits a call. That is a sensible trade for free text you intend to store or share (ticket archives, datasets for testing, text sent on to another system), where one processor seeing it briefly beats every downstream system keeping it. It is the wrong tool for structured log records, where a local redact() by field name is free, faster and keeps the data in your process. And pattern matching has limits in both directions: it will not recognise a name or a street address on its own, so it complements field-level redaction rather than replacing it.
A worker logs the full Email Validator response for every signup to help with debugging, and caches it keyed by the email address for 30 days. Which change removes the personal data while keeping the debugging and caching value?
Retention is a property of the store
A retention policy that lives in a document is a wish. One that lives in the storage layer is a fact. Classify each kind of data your system holds and give each class a lifetime that the store enforces on its own: TTLs on cache keys, lifecycle rules on log buckets, scheduled deletion jobs with alerts when they fail.
| Data class | Example | Typical lifetime | Enforced by |
|---|---|---|---|
| Raw request payloads | The full signup form sent to a worker | Until the job completes | Delete on ack; never logged |
| Projected API results | { isValid, isDisposable } per email token | Hours to weeks, by freshness need | Cache TTL |
| Operational logs | Redacted request and error events | Days to weeks | Log store lifecycle rule |
| Audit records | Sanctions screen outcomes, consent changes | Set by legal obligation, often years | Append-only store with its own policy |
| Derived analytics | Counts by country and day | Long; no personal data left | Aggregation before storage |
Audit records sit apart for a reason: they are the one class you keep because of an obligation, so they must not be swept away by the short lifetimes that suit everything else, and they must not become a back door for keeping operational data longer than you should.
Privacy rules as engineering requirements
Regimes such as the EU's GDPR and California's CCPA differ in scope and detail, and which ones apply to you is a legal question. Several of their recurring ideas translate directly into requirements an architecture can meet or fail:
- Know where personal data goes. Keep a data map: each personal field, each store it lands in, and each third party it is sent to. The outbound contract above is the third-party half of that map. Every API provider that receives personal data from you is processing it on your behalf, and a review will expect you to have read their terms on it.
- Purpose limitation. Data collected to verify a signup should not quietly become marketing data. Tag stores with their purpose and make reuse a deliberate decision.
- Access and deletion. People can ask what you hold about them and, with exceptions, ask you to delete it. That is only answerable if every store is findable by a subject identifier, including caches and logs. Pseudonymous tokens help: deletion becomes "recompute the token, delete everywhere it appears".
- Security of processing. Encryption in transit and at rest, least-privilege access to logs and audit stores, and scoped API keys so a leaked credential reaches only what it must.
- Processors and transfers. Know where each provider processes data and on what terms. For APIVerve, the security page states that request and response content is not stored beyond what real-time processing needs, that usage logs are kept for 90 days, and that processing runs on Google Cloud, primarily in US regions. A data processing agreement is published. Read those documents as part of your own review rather than relying on a summary, including this one.
Sanctions screening as a regulatory control
Most checks in your system are business decisions: you can choose how strict an email check is. A sanctions screen is different. Where it applies, it is a control your business is expected to operate, and the test of a control is not only "did it run?" but "can you show it ran, on what, against which lists, and what you did with the result?" The sanctions screening lesson covered the call and the fuzzy-match review. At architecture level, four properties matter:
- Fail closed. A screen that errors, times out or hits an open circuit leaves the case
pending, neverclear. This is lesson one's posture, applied where it matters most. - Record what was actually screened. Store
data.listsfrom the response, with each list'spublishDate, not the lists your code asked for. - Re-screen on change and on schedule. Lists change; your customers' details change. A screen at signup alone does not cover a listing added next month.
- Separate the audit trail from operational logs. Different store, different access, different lifetime.
curl "https://api.apiverve.com/v1/sanctionscreening?name=Banco+Nacional+de+Cuba" \
-H "x-api-key: YOUR_API_KEY"verdict is exact. Compare data.lists (what was screened, with publish dates) with the lists you would have recorded from your own configuration.
An audit trail that proves something
An audit record answers who, what, when, with what input, with what result, and under which version of the rules. To be evidence rather than a log, it also needs to be append-only and tamper-evident. A simple, effective technique is a hash chain: each entry includes a hash of the previous one, so editing or deleting any entry breaks every hash after it.
import { createHash } from 'node:crypto';
import { token } from './pseudonym.mjs';
let prevHash = 'genesis'; // in production: read the last entry's hash from the store
export function auditEntry({ actor, partyId, name, result, rulesVersion }) {
const entry = {
at: new Date().toISOString(),
actor, // service or reviewer id
control: 'sanctions-screen',
partyId,
nameScreened: name, // the input is the evidence; keep it here, not in debug logs
inputToken: token(name), // joinable without exposing the name elsewhere
outcome: result.outcome, // clear | review | pending
verdict: result.verdict ?? null,
lists: (result.lists ?? []).map((l) => ({ code: l.code, publishDate: l.publishDate })),
rulesVersion,
prevHash,
};
entry.hash = createHash('sha256').update(JSON.stringify(entry)).digest('hex');
prevHash = entry.hash;
return entry; // append to a write-once store; never update in place
}The audit record keeps the screened name because the name is the evidence: an auditor needs to see what was checked. That is exactly why it lives in a separate store with narrow access and its own retention, while the operational logs for the same request carry only inputToken.
Which property makes a sanctions audit trail evidence rather than just a log?
What you have now
Outbound calls carry only their contracted fields. Responses are projected before they persist. Logs and traces pass through one redaction function, caches are keyed by pseudonymous tokens, and every data class has a lifetime its store enforces. Regulatory controls fail closed and leave a tamper-evident trail.
The last lesson pulls all four lessons of this course into a single document: the architecture review that a multi-API system should pass before it carries real traffic. Next: Write an architecture review.