APIVerveAcademy
Start for free
APV-400Architect Agent Platforms
Lesson 4 of 4

Evaluate and govern AI agents in production

Build replay evals and regression gates, trace every agent run, measure cost per run and per outcome, detect drift and set up ownership and change control.

  • 50 min
  • Expert

At the Associate level you built an eval for one agent: a case set, recorded API responses, a few runs per case and a pass rate you watched by eye. That is the right foundation, and on a platform it stops being enough for a simple reason: nobody is watching by eye. Twenty deployments change prompts, models, tool manifests and policies on different schedules, and the person merging a prompt edit to the support agent is not going to notice that it broke a case they have never read.

This lesson turns evaluation and oversight into machinery. You will make replay evals strict enough to trust, put a regression gate in front of every change, trace runs so any one can be reconstructed, measure cost per outcome rather than per call, watch for four kinds of drift, and give every agent an owner and a manifest that says exactly what it is.

Replay evals you can trust

A replay eval runs the real agent loop against recorded tool responses, so repeated runs cost model tokens but no API credits, and a failure reflects the agent rather than live data changing underneath it. Three refinements make it reliable enough to gate on.

Record against a manifest. A fixture file is only meaningful for the tool definitions it was recorded with. Store the deployment's tool manifest hash (lesson 1) in the fixture file, and refuse to replay when the current manifest differs. A changed tool description is a change to the agent, and the eval must say so rather than quietly passing on old assumptions.

Make unrecorded calls fail loudly. When the agent calls a tool with arguments that have no recording, the replay must not fall through to the live API, and it must not pretend the API said "not found". Report it as its own failure kind, unrecorded. That separates "the agent wandered somewhere new" from "the agent handled a genuine no-result correctly", which are opposite findings.

Record failures too. The error paths are where agents misbehave, so the case set needs recorded 400s and 404s, not just successes. Record the envelope and status as they came back.

Recording is the only step that costs credits. A phone validation case, for example, records exactly what this returns:

Try it livePhone Number Validator10 credits
GEThttps://api.apiverve.com/v1/phonenumbervalidator
curl "https://api.apiverve.com/v1/phonenumbervalidator?number=%2B14155552671" \
  -H "x-api-key: YOUR_API_KEY"
Sign in or create a free account to run this with your own key.

Recorded once, replayed hundreds of times. riskScore and riskLevel are premium fields: record on the plan production runs on, or the fixture will not match what agents see.

replay.mjs
// Node 18+, ESM. Strict replay: keyed by tool and canonical arguments, pinned to a manifest hash.
import { readFileSync } from 'node:fs';

const canonical = (args) => JSON.stringify(Object.keys(args ?? {}).sort().map((k) => [k, String(args[k]).trim()]));
export const fixtureKey = (tool, args) => `${tool}:${canonical(args)}`;

export function loadReplay(path, manifest) {
  const fx = JSON.parse(readFileSync(path, 'utf8'));
  if (fx.manifestHash !== manifest.hash) {
    throw new Error(`Fixtures recorded for tools ${fx.manifestHash}, deployment now uses ${manifest.hash}. Re-record.`);
  }
  return async function replay(tool, args) {
    const hit = fx.responses[fixtureKey(tool, args)];
    if (!hit) {
      return { isError: true, unrecorded: true, content: [{ type: 'text', text: `${tool}: no recording for these arguments` }] };
    }
    return hit.status === 200
      ? { isError: false, billed: true, content: [{ type: 'text', text: JSON.stringify(hit.body.data) }], structuredContent: hit.body }
      : { isError: true, content: [{ type: 'text', text: `${tool} failed (${hit.status}): ${JSON.stringify(hit.body.error)}` }] };
  };
}

Score each run in layers, cheapest and most objective first:

  1. Trajectory. Which tools were called, with which arguments, in what order where order matters. Run every recorded call's arguments through checkArgs from lesson 2 and fail any that would have been rejected.
  2. Policy. Replay the policy function from lesson 3 over every call. A case built around a payout to an unknown IBAN expects the decision approve; allow is a failure even if the final answer looks fine.
  3. Budget. Sum the credits from the definition prices. A run that reached the right answer for three times the usual spend is a regression.
  4. Outcome. The final answer or structured result falls inside the acceptable set you wrote down before running anything.

Keep model-graded checks for the last layer only, for qualities code cannot judge, and grade them against a written rubric. The first three layers are deterministic and fast, and they catch most regressions.

Regression gates on every change

An eval nobody has to pass is a report. A gate is an eval that blocks a merge. On a platform, anything that changes what an agent sees or may do goes through the gate: the system prompt, the model id, the tool manifest, the policy file, the budgets and the fixture set itself.

Run every case several times, because a single pass from a probabilistic system tells you little. Five runs per case is a common starting point; flaky cases need more. Store the result as a baseline, and compare each candidate against it with explicit rules:

gate.mjs
// Node 18+, ESM. Compare a candidate eval run against the stored baseline. Exit 1 blocks the merge.
import { readFileSync } from 'node:fs';

const base = JSON.parse(readFileSync('evals/baseline.json', 'utf8'));   // { cases: { id: { pass, runs, credits } } }
const cand = JSON.parse(readFileSync('evals/candidate.json', 'utf8'));
const rate = (c) => c.pass / c.runs;
const failures = [];

for (const [id, b] of Object.entries(base.cases)) {
  const c = cand.cases[id];
  if (!c) { failures.push(`${id}: missing from candidate run`); continue; }
  if (c.policyViolations) failures.push(`${id}: ${c.policyViolations} policy violation(s)`);   // never tolerated
  if (c.unrecorded) failures.push(`${id}: ${c.unrecorded} unrecorded tool call(s)`);
  if (rate(b) === 1 && rate(c) < 0.8) failures.push(`${id}: stable case fell to ${c.pass}/${c.runs}`);
  if (c.credits / c.runs > (b.credits / b.runs) * 1.15) failures.push(`${id}: credits per run up more than 15%`);
}
const total = (s) => Object.values(s.cases).reduce((a, c) => [a[0] + c.pass, a[1] + c.runs], [0, 0]);
const [bp, br] = total(base); const [cp, cr] = total(cand);
if (cp / cr < bp / br - 0.03) failures.push(`overall pass rate ${(cp / cr).toFixed(3)} vs baseline ${(bp / br).toFixed(3)}`);

if (failures.length) { console.error(failures.join('\n')); process.exit(1); }
console.log(`gate passed: ${cp}/${cr} runs`);

The thresholds are judgement calls you write down once rather than argue about per pull request. The structure matters more than the numbers. Policy violations and unrecorded calls have zero tolerance, because each is a qualitative change in what the agent does. Stable cases are protected individually, so a change cannot trade a solid case for a marginal improvement elsewhere. Cost is a gated metric, not a dashboard curiosity. And when a change is meant to alter behaviour, the baseline is updated in the same pull request, so the reviewer sees the expected difference rather than a mysterious green tick.

Check yourself

A prompt edit raises the overall pass rate from 88% to 92%, but one case that passed 5/5 in the baseline now passes 3/5. How should the gate above treat it?

Run every example on your own key. Free plan, no card.

Get a free API key→All courses

Scaling up?

Volume pricing, custom SLAs, and dedicated support for high-traffic teams.

Contact sales