Every guardrail in this lesson rests on one assumption: the model will sometimes be wrong, and sometimes it will have been misled on purpose. Not often, but at platform volume "not often" is a schedule. A control that depends on the model choosing to behave is not a control, so everything here lives in code the model cannot talk to.
The Associate courses introduced the pieces for one agent: a propose-then-approve pattern, a closure that counted credits, delimiters around untrusted text. This lesson turns them into platform infrastructure that every deployment inherits: a classification of every tool by consequence, an approval queue that cannot be replayed or redirected, a policy function that runs on every call, budgets enforced at several scopes, and a way to contain prompt injection that does not depend on spotting it.
Classify every tool by consequence
Before you can decide what needs a human, you need a vocabulary. Give every tool in the platform registry exactly one tier, and make the tier part of the tool's definition in code, not a comment.
| Tier | What it does | Default handling | Examples |
|---|---|---|---|
read | Looks something up or computes; no outbound side effects | Runs automatically, within budget | Phone, IBAN and email validation, number formatting |
fetch | Reads content from an address the arguments choose | Runs if the host is allowed; output is untrusted | URL to Markdown, Website to Text, screenshots |
write | Changes state you can reverse | Runs automatically only for an untainted run; otherwise approval | Tag a ticket, add a CRM note |
external | Contacts a person, moves money or cannot be undone | Always a proposal for a human | Send an email, issue a payout, lock an account |
forbidden | Exists in the codebase but no agent may call it | Refused | Change a key, raise a budget, export a customer list |
Every APIVerve endpoint is in the first two tiers. The MCP server marks every tool readOnlyHint: true, and none of them changes state in your account. But a tool that fetches a URL makes a request to an address the model chose, which is an outbound channel: a URL with data in its query string sends that data to whoever runs the server. That is why fetch is its own tier and not part of read.
The forbidden tier is there for a reason. Code that can adjust a budget or a scope will eventually be wrapped as a tool by someone trying to be helpful. Naming the tier makes the review question explicit.
Approval queues that hold up
The Associate version of an approval queue recorded a proposal and let an admin route run it. That is the right shape. At platform scale it needs five more properties, each closing a specific failure.
- Bind the approval to the exact arguments. Store a hash of the tool name and canonicalised arguments. At execution, recompute it. If anything changed between what the reviewer saw and what will run, refuse.
- Separate the requester from the approver. The agent's deployment identity can propose; only a human identity can decide, and never the same human whose request started the run if your policy requires two people.
- Expire proposals. A payout approved next Tuesday on last Monday's evidence is not the decision anyone made. Pending proposals lapse.
- Execute exactly once. Move the proposal from
approvedtoexecutingwith a compare-and-set, and pass the proposal id as the idempotency key to whatever performs the action. A double-clicked approve button or a retried worker then runs the action once. - Attach evidence and re-check policy at execution. The reviewer sees the tool results that justify the action, and the policy function (next section) runs again at execution time, because limits and account state change while a proposal waits.
Evidence is where APIVerve's verification APIs earn their place. A payout agent proposing a transfer should attach a validation of the destination account, so the reviewer is not judging a raw string:
curl "https://api.apiverve.com/v1/ibanvalidator?iban=GB82WEST12345698765432" \
-H "x-api-key: YOUR_API_KEY"isValid, countryCode and the checks object are what a reviewer needs beside the proposal. Bank and account breakdown fields are premium and may be null.
// Node 18+, ESM. An approval queue bound to exact arguments, with expiry and single execution.
import { createHash, randomUUID } from 'node:crypto';
const canonical = (v) =>
Array.isArray(v) ? v.map(canonical)
: v && typeof v === 'object' ? Object.fromEntries(Object.keys(v).sort().map((k) => [k, canonical(v[k])]))
: v;
const digest = (tool, args) => createHash('sha256').update(JSON.stringify([tool, canonical(args)])).digest('hex');
const TTL_MS = 4 * 60 * 60 * 1000;
export const queue = new Map(); // a table with a unique id and conditional updates in production
export function propose(ctx, tool, args, evidence) {
const id = randomUUID();
queue.set(id, {
id, tool, args, evidence, hash: digest(tool, args), status: 'pending',
tenantId: ctx.tenantId, requestedBy: ctx.deploymentId, startedBy: ctx.userId, runId: ctx.runId,
expiresAt: Date.now() + TTL_MS,
});
return { isError: false, content: [{ type: 'text', text: `Proposed ${tool} as ${id}. A person will decide; do not propose it again.` }] };
}
export function decide(id, reviewer, approve) {
const p = queue.get(id);
if (!p || p.status !== 'pending') throw new Error('No pending proposal with that id');
if (Date.now() > p.expiresAt) { p.status = 'expired'; throw new Error('Proposal expired'); }
if (!reviewer.isHuman || reviewer.id === p.startedBy) throw new Error('Whoever started the run cannot approve it');
Object.assign(p, { status: approve ? 'approved' : 'declined', reviewer: reviewer.id, decidedAt: Date.now() });
}
export async function executeApproved(id, handlers, policy) {
const p = queue.get(id);
if (!p || p.status !== 'approved') throw new Error('Not approved');
p.status = 'executing'; // compare-and-set in your database
if (digest(p.tool, p.args) !== p.hash) { p.status = 'failed'; throw new Error('Arguments changed after approval'); }
const verdict = policy({ tenantId: p.tenantId, atExecution: true }, { name: p.tool, args: p.args });
if (verdict.decision === 'deny') { p.status = 'failed'; throw new Error(`Policy now denies: ${verdict.reason}`); }
const result = await handlers[p.tool](p.args, { idempotencyKey: p.id });
Object.assign(p, { status: 'done', executedAt: Date.now() });
return result;
}Two habits keep the queue useful. Reviewers see the evidence and the run's trace, not a one-line summary the agent wrote about itself, because the agent is the thing being checked. And there is no "approve all" button. If reviewers want one, the agent proposes too much, and the fix is a better policy, not a faster click.
A reviewer approves a payout proposal for 1,200 EUR. Before it executes, a bug in the agent's retry path updates the stored arguments to 12,000 EUR. Which property of the queue above stops the transfer?
Policy checks belong in code
A system prompt that says "never pay more than 5,000 EUR without approval" is a request. A function that returns approve for any payout above 5,000 EUR is a rule. Put every rule you would be embarrassed to see broken into one policy function that every tool call passes through, after argument checking and before execution.
// Node 18+, ESM. One decision per call: allow, approve or deny. Pure, so it is easy to test.
export function policy(ctx, call, registry, limits) {
const tool = registry.get(call.name);
if (!tool || !ctx.allowedTools?.has(call.name)) return { decision: 'deny', reason: `${call.name} is not enabled for this deployment` };
if (tool.tier === 'forbidden') return { decision: 'deny', reason: `${call.name} may not be called by agents` };
if (tool.tier === 'fetch') {
let u;
try { u = new URL(call.args.url); } catch { return { decision: 'deny', reason: 'url is not a valid absolute URL' }; }
if (u.protocol !== 'https:' || !limits.fetchHosts.has(u.hostname)) return { decision: 'deny', reason: `${u.hostname} is not an allowed host` };
if (u.search) return { decision: 'deny', reason: 'query strings are not allowed on fetched URLs' };
}
if (call.name === 'issue_payout') {
if (call.args.amount > limits.payoutMax) return { decision: 'deny', reason: `above the ${limits.payoutMax} hard limit` };
if (!limits.knownIbans.has(call.args.iban)) return { decision: 'approve', reason: 'destination is not on file for this vendor' };
}
if (tool.tier === 'external') return { decision: 'approve', reason: 'external actions always need a person' };
if (tool.tier === 'write' && ctx.tainted) return { decision: 'approve', reason: 'run has read untrusted content' };
return { decision: 'allow' };
}The policy is pure: it reads the call and context and returns a decision. That makes it trivially testable, and lesson 4's eval harness runs it on every recorded case. It also makes it reviewable: the full list of what agents may do is one file, owned by someone, changed by pull request.
The loop handles the three outcomes uniformly. allow runs the call. approve routes it to propose and tells the model the action is pending. deny returns an isError result with the reason, so the model can choose another path or explain to the user, and records the denial in the trace. A spike in denials for one deployment is one of the most useful alerts you can have: either the agent is confused or something is steering it.
Per-run budgets at several scopes
An agent run needs a ceiling on credits and on calls, enforced before each call. On a platform, one ceiling is not enough: you also want a per-tenant daily limit and perhaps a per-deployment monthly one, and a run must stop if any of them would be crossed. The cleanest way to get that is a pure guard function that knows nothing about scopes, applied once per scope.
// Node 18+, ESM. A per-run budget guard on credits and call count.
// budget: { maxCredits, maxCalls } spent: { credits, calls } tool: { name, credits }
export function allowCall(budget, spent, tool) {
const cost = tool?.credits;
if (typeof cost !== 'number' || !Number.isFinite(cost) || cost < 0) {
return { ok: false, reason: `no known price for ${tool?.name ?? 'this tool'}` }; // fail closed
}
if (spent.calls + 1 > budget.maxCalls) {
return { ok: false, reason: `call limit reached (${spent.calls}/${budget.maxCalls})` };
}
if (spent.credits + cost > budget.maxCredits) {
return { ok: false, reason: `credit budget would be exceeded (${spent.credits} + ${cost} > ${budget.maxCredits})` };
}
return { ok: true };
}The boundaries are deliberate. A call that brings spending to exactly the budget is allowed; one that would pass it is not. The call count is checked before credits, so a loop of cheap calls is caught even when it costs almost nothing. And a tool with no known price is refused rather than assumed cheap, which matters when someone adds a tool to the registry and forgets its cost. The price comes from the endpoint's definition (tokenUse, the same figure the MCP server appends to each tool description), not from a response header: x-api-credits-used is the account's running total for the cycle, not the cost of the call that returned it.
Now apply it at every scope, reserve before the call, and settle after:
// Node 18+, ESM. Reserve at every scope, call, then release what was not billed.
import { allowCall } from './budget.mjs';
export async function guardedCall(scopes, tool, run) {
// scopes: [{ name: 'run', budget, spent }, { name: 'tenant-day', budget, spent }, ...]
for (const s of scopes) {
const v = allowCall(s.budget, s.spent, tool);
if (!v.ok) {
return { isError: true, content: [{ type: 'text', text: `Budget (${s.name}): ${v.reason}. Stop calling tools and answer with what you have.` }] };
}
}
for (const s of scopes) { s.spent.calls += 1; s.spent.credits += tool.credits; } // reserve
const result = await run();
if (result.isError) {
for (const s of scopes) s.spent.credits -= tool.credits; // APIVerve bills only successful calls
}
return result;
}Three platform details make this hold under real conditions. Reservation must be atomic across processes: when two runs for the same tenant check the daily scope at once, both can pass a check-then-write. Store spent for shared scopes in your database and reserve with a conditional increment. The call count is not released on failure, because a run that fails repeatedly is exactly what the call cap exists to stop. And spent lives in the run record, not in a closure, so a run resumed by another worker after a crash continues with the budget it had, instead of starting fresh.
A run has budget { maxCredits: 60, maxCalls: 5 } and has spent { credits: 40, calls: 3 }. The model calls a tool priced at 20 credits. What does allowCall return?
Prompt injection through tool outputs
Any tool that returns text someone else wrote is a channel through which that someone can address your agent. Web pages are the obvious case:
curl -X POST "https://api.apiverve.com/v1/urltomarkdown" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com"}'Harmless page, but every character of the Markdown was written by whoever controls the site, including anything hidden from human readers.
The Associate course established that you cannot filter your way out of this, because an instruction is just a sentence, and that delimiters and allow-lists reduce the risk without removing it. At platform level, two architectural patterns change the shape of the problem rather than the odds.
Taint tracking. Mark tools whose output is untrusted: every fetch tool, anything returning email bodies, tickets, documents or user uploads. The moment such a tool returns in a run, the run is tainted for the rest of its life. The policy function above already reads ctx.tainted: in a tainted run, reversible writes need approval, and fetches are limited to the allow-list with no query strings. You do not need to detect injection; you assume it happened and reduce what it can do.
Quarantined extraction. When a run genuinely needs facts from untrusted content, such as the amount and destination on a supplier's invoice page, do not let the planning model read the page at all. A separate model call, with no tools, reads the content and must return JSON matching a strict schema. Your code validates that JSON with the same checkArgs from lesson 2, and only the validated fields reach the planner. An injected paragraph can corrupt a field's value, but it cannot add a tool call, change the plan's instructions or smuggle prose into the planner's context.
Notice what quarantine does not protect: the values. An attacker who controls the invoice page controls the IBAN on it. That is why the policy function compares the extracted IBAN against the vendor's account on file and routes any mismatch to a human, and why the reviewer's evidence includes an independent validation of the account. Injection defence ends in value-level policy, not in the model.
Least privilege, end to end
Least privilege is not one setting. It is the same idea applied at every layer, so that a failure at one layer is caught by the next.
| Layer | Narrowest setting |
|---|---|
| Key | A sub-key per deployment, blocked from every endpoint outside its job; an IP allow-list for server-side agents on Mega |
| Tool surface | The deployment's pinned manifest; single-tool MCP paths for hosted use |
| Run | Only the tools this task needs, chosen by code from the task type, not the whole deployment set |
| Arguments | checkArgs on every call; tenant and key from the session, never from arguments |
| Actions | Consequence tiers, the policy function and the approval queue |
| Spend | allowCall at run, tenant and deployment scope |
| Data | Tenant-scoped references, expiry on stored outputs, pseudonymised logs |
Read the table from the bottom when something goes wrong. An injected instruction that gets past the model meets the policy function; one that gets past a policy bug meets the budget; a key that leaks meets its scoping and its revocation. None of these layers is sufficient alone, and none depends on the model.
Next
You can now bound what every agent on the platform may do, spend and read. The last question is how you know it is working, and how you change it safely: Evaluate and govern agents in production.