How to Test AI Agents: Evals, Guardrails and Approval

You have two working agents. "Working" so far means you ran each a few times and liked the output. That is how every agent starts, and it is not enough to ship on, for two reasons.

The first is that agents are not deterministic. The same input can produce a different sequence of tool calls and a different recommendation on the next run. One good run tells you the agent can get it right, not how often it does.

The second is that agents change underneath you. You edit a line of the system prompt to fix one case and quietly break three others. Your provider releases a new model version. You add a tool and the agent starts reaching for it when it should not. Without a test set you will find out from users.

This lesson builds the test set and harness for the signup triage agent, then adds the guards that hold when the tests miss something: a tool allow-list with input validation, a credit budget, approval gates for anything consequential, and logs you can actually read later. Everything here applies to the research agent too.

Make the agent testable first

The triage agent calls APIVerve through executeApiTool and logs through the default onEvent. Both need to be swappable, so a test can feed recorded responses in and capture every tool call that comes out. Change two lines in triage.js:

triage.js (changes only)

// 1. Accept the API caller and event hook as options.
export async function triage(signup, { callApi = executeApiTool, onEvent } = {}) {

// 2. In execute(), call the API through the option instead of directly:
//      const r = await callApi(name, input);
// 3. Pass the hook through to the loop:
//      const run = await runAgent({ system: SYSTEM, prompt, execute, maxTurns: 6, onEvent,
//        tools: [...apiTools(SIGNAL_APIS), ...LOCAL_TOOLS] });

When onEvent is undefined, runAgent falls back to its default logger, so production behaviour does not change.

Write a case set, then record the world once

A test case is an input plus what an acceptable outcome looks like. Write it down before you run anything, while your judgement is not anchored by what the agent happened to say.

cases.json

[
  {
    "id": "disposable-email",
    "signup": { "signupId": "t1", "email": "[email protected]", "ip": "173.172.81.20", "flaggedBecause": "burst of signups from one domain" },
    "expect": { "recommendation": ["reject", "request_verification"], "mustCall": ["emailvalidator"], "maxApiCalls": 3 }
  },
  {
    "id": "repeat-trial",
    "signup": { "signupId": "t2", "email": "[email protected]", "ip": "103.62.49.210", "flaggedBecause": "datacenter IP" },
    "expect": { "recommendation": ["reject", "request_verification"], "mustCall": ["emailvalidator", "find_accounts"], "maxApiCalls": 3 }
  },
  {
    "id": "role-account-residential",
    "signup": { "signupId": "t3", "email": "[email protected]", "ip": "173.172.81.20", "flaggedBecause": "role address" },
    "expect": { "recommendation": ["approve", "request_verification"], "mustCall": ["emailvalidator"], "maxApiCalls": 3 }
  },
  {
    "id": "clean-email-datacenter-ip",
    "signup": { "signupId": "t4", "email": "[email protected]", "ip": "103.62.49.210", "flaggedBecause": "datacenter IP" },
    "expect": { "recommendation": ["approve", "request_verification"], "mustCall": ["emailvalidator", "vpndetector"], "maxApiCalls": 3 }
  }
]

Notice what the expectations are. Not "recommends reject", but "recommends one of these". For a clean address on a datacenter IP, approve and request_verification are both defensible; reject is not. Writing acceptable sets instead of single answers stops your tests from failing on reasonable judgement calls, so that when one does fail, it means something.

The first case also encodes a trap worth showing. Validate the disposable address:

Try it liveEmail Validator20 credits

GEThttps://api.apiverve.com/v1/emailvalidator

1
2

curl "https://api.apiverve.com/v1/emailvalidator?email=test%40mailinator.com" \
  -H "x-api-key: YOUR_API_KEY"

isValid is true AND isDisposable is true. A deliverable throwaway address is still a throwaway address.

isValid is true: the address is well-formed and the domain receives mail. An agent that reads isValid as "fine" will approve it. The test catches that regression whenever a prompt edit makes the agent lazier about which field it reads.

Now the cost problem. Four cases, three runs each, three API calls per run: 36 calls at 10 to 20 credits each, every time you change a comma in the prompt. And because live data drifts, a test that fails today might be the IP database updating, not your agent getting worse.

So record the world once and replay it. Make each case's API calls for real, save the responses, and serve them from the file during tests:

eval.js

import { readFileSync, writeFileSync } from 'node:fs';
import { triage } from './triage.js';
import { DEFS, callApiverve } from './tools.js';
import { guard } from './guard.js';

const cases = JSON.parse(readFileSync(new URL('./cases.json', import.meta.url), 'utf8'));
const FIXTURES = new URL('./fixtures.json', import.meta.url);
const key = (name, input) =>
  `${name}:${JSON.stringify(Object.keys(input).sort().map((k) => [k, String(input[k]).trim().toLowerCase()]))}`;

if (process.argv[2] === 'record') {
  const fixtures = {};
  for (const c of cases) {
    for (const [name, input] of [['emailvalidator', { email: c.signup.email }], ['vpndetector', { ip: c.signup.ip }], ['iplookup', { ip: c.signup.ip }]]) {
      const r = await callApiverve(DEFS.get(name), input);
      if (!r.ok) throw new Error(`${name} failed while recording: ${r.error}`);
      fixtures[key(name, input)] = r.data;
    }
  }
  writeFileSync(FIXTURES, JSON.stringify(fixtures, null, 2));
  console.log(`Recorded ${Object.keys(fixtures).length} responses.`);
  process.exit(0);
}

const fixtures = JSON.parse(readFileSync(FIXTURES, 'utf8'));
async function replay(name, input) {
  const data = fixtures[key(name, input)];
  return data
    ? { content: JSON.stringify(data), isError: false }
    : { content: `${name} failed (404): no result for this input`, isError: true };
}

const API_TOOLS = new Set(['emailvalidator', 'vpndetector', 'iplookup']);

function score(c, result, calls) {
  const f = [];
  const names = calls.map((e) => e.name);
  for (const t of c.expect.mustCall) if (!names.includes(t)) f.push(`never called ${t}`);
  for (const e of calls) {
    if (e.name === 'emailvalidator' && String(e.input.email).toLowerCase() !== c.signup.email.toLowerCase()) f.push(`emailvalidator got ${e.input.email}`);
    if ((e.name === 'vpndetector' || e.name === 'iplookup') && e.input.ip !== c.signup.ip) f.push(`${e.name} got ${e.input.ip}`);
    if (e.isError) f.push(`${e.name} returned an error`);
  }
  const apiCalls = calls.filter((e) => API_TOOLS.has(e.name)).length;
  if (apiCalls > c.expect.maxApiCalls) f.push(`${apiCalls} API calls, max ${c.expect.maxApiCalls}`);
  const rec = result.recommendation?.recommendation;
  if (!c.expect.recommendation.includes(rec)) f.push(`recommended ${rec ?? 'nothing'}`);
  return f.concat(result.problems);
}

const RUNS = Number(process.env.RUNS || 3);
let passed = 0;
for (const c of cases) {
  for (let i = 1; i <= RUNS; i++) {
    const calls = [];
    const callApi = guard(replay, { allow: [...API_TOOLS], creditBudget: 60 });
    const result = await triage(c.signup, { callApi, onEvent: (e) => e.type === 'tool' && calls.push(e) });
    const failures = score(c, result, calls);
    if (!failures.length) passed++;
    console.log(`${failures.length ? 'FAIL' : 'pass'}  ${c.id} #${i}${failures.length ? `  ${failures.join('; ')}` : ''}`);
  }
}
console.log(`\n${passed}/${cases.length * RUNS} runs passed`);

1
2

node eval.js record   # once: real API calls, about 160 credits for these four cases
node eval.js          # as often as you like: zero API credits, model tokens only

The replay also checks arguments for free. If the agent calls vpndetector with a mistyped IP, or with the email address in the ip field, there is no recorded response for that input, the call returns an error, and the run fails with a message saying exactly which argument was wrong. The explicit checks in score make the same point readable in the output.

Run each case several times. A case that passes three runs out of three is solid; two out of three is a coin you have not finished flipping. Track the overall pass rate per change. If a prompt edit takes you from 12 out of 12 to 10 out of 12, you have learned something before your users did.

Check yourself

Why replay recorded API responses in the eval instead of calling the live APIs each time?

What else belongs in a case set

Four cases is a start. Grow it from three sources. Every production disagreement: when a reviewer overrules the agent, that signup becomes a case with the reviewer's decision as the expectation. Every incident: whatever went wrong becomes a case that fails until it is fixed. Adversarial inputs: for the research agent, a page that says "ignore previous instructions" in its body; for triage, an email address built to look like an instruction. The expected outcome for these is that the agent behaves exactly as it would on a boring input.

Guard the tools, not just the prompt

Tests tell you how often the agent is right. Guards decide what happens when it is wrong. They live in code, between the loop and the APIs, because a system prompt is a request and code is a rule.

guard.js

import { isIP } from 'node:net';
import { DEFS } from './tools.js';

const VALIDATORS = {
  email: (v) => typeof v === 'string' && v.length <= 254 && /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(v),
  ip: (v) => typeof v === 'string' && isIP(v) !== 0,
};

// Wraps an API caller with an allow-list, input checks, de-duplication, per-tool limits and a credit budget.
export function guard(callApi, { allow, creditBudget, maxCallsPerTool = 2 }) {
  let spent = 0;
  const counts = new Map();
  const cache = new Map();

  return async (name, input) => {
    if (!allow.includes(name)) return { content: `Tool ${name} is not available.`, isError: true };
    for (const [k, v] of Object.entries(input ?? {})) {
      if (VALIDATORS[k] && !VALIDATORS[k](v)) return { content: `Invalid ${k}: ${JSON.stringify(v)}`, isError: true };
    }

    const cacheKey = `${name}:${JSON.stringify(input)}`;
    if (cache.has(cacheKey)) return cache.get(cacheKey);

    const n = (counts.get(name) ?? 0) + 1;
    if (n > maxCallsPerTool) return { content: `${name} already called ${maxCallsPerTool} times. Use the results you have.`, isError: true };
    const cost = DEFS.get(name)?.tokenUse ?? 0;
    if (spent + cost > creditBudget) return { content: `Credit budget of ${creditBudget} reached. Submit with the evidence you have.`, isError: true };

    counts.set(name, n);
    spent += cost;
    const r = await callApi(name, input);
    cache.set(cacheKey, r);
    return r;
  };
}

In server.js, create a fresh guard per review, because the budget is per run:

server.js (inside POST /flagged)

1
2

const callApi = guard(executeApiTool, { allow: ['emailvalidator', 'vpndetector', 'iplookup'], creditBudget: 60 });
const result = await triage({ signupId, email, ip, flaggedBecause, createdAt: new Date().toISOString() }, { callApi });

Each check earns its place:

The allow-list duplicates what the tool list already implies, deliberately. Tool lists get edited; a guard in front of the executor still says no.
Input validation rejects an IP that is not an IP before it costs a call. The rejection is a readable error, so the model can correct itself.
De-duplication returns the stored result when the agent asks the same question twice, which agents do, especially after an error. The repeat costs nothing.
The credit budget turns "the agent got stuck in a loop" from an invoice into a log line. Sixty credits covers the three checks this agent needs with room for one retry. When the budget runs out, the agent is told to submit with what it has, and maxTurns from lesson 1 catches it if it does not.

Behind all of this sits the outermost guard, which you set up once in the dashboard: the agent's key is a sub-key, scoped to the APIs it uses. If everything above fails, the key itself refuses.

Try it liveVPN Proxy Detector10 credits

GEThttps://api.apiverve.com/v1/vpndetector

1
2

curl "https://api.apiverve.com/v1/vpndetector?ip=103.62.49.210" \
  -H "x-api-key: YOUR_API_KEY"

A valid IP passes the validator. 'not-an-ip' would be rejected by guard.js before any credits are spent.

Human approval for anything consequential

The triage agent has no consequential tools: every tool reads. That is the best position to be in, and you should keep agents there for as long as you can. Sooner or later someone will want the agent to do something, send the verification email, lock the account, issue the refund, and the rule is simple to state: any tool that changes state outside the agent, spends money, contacts a person or cannot be undone becomes a proposal, not an action.

approvals.js

import { randomUUID } from 'node:crypto';

export const approvals = new Map(); // use your database in production

// What the agent's tool executor calls. Nothing happens yet.
export function propose(action, input, context) {
  const id = randomUUID();
  approvals.set(id, { id, action, input, context, status: 'pending', proposedAt: new Date().toISOString() });
  return { content: `Proposed ${action} as ${id}. A person will decide. Do not propose it again.`, isError: false };
}

// What your admin route calls. The only place an action runs.
export async function approve(id, reviewer, handlers) {
  const a = approvals.get(id);
  if (!a || a.status !== 'pending') throw new Error('No pending approval with that id');
  Object.assign(a, { status: 'approved', reviewer, approvedAt: new Date().toISOString() });
  return handlers[a.action](a.input);
}

The agent sees a tool called, say, send_verification_email. Its executor calls propose, which records the request and returns a message saying it is pending. The agent's run finishes. Later a person opens the approvals list, reads the context, and either approves, which runs the real handler under their name, or declines. The agent never holds a function that sends email; the approval code does.

Two mistakes to avoid. Do not let the agent's message decide the risk level ("this is low risk, auto-approve"): the agent is the thing being checked. And do not give the approval route a bulk "approve all" button. If reviewers need one, the agent is proposing too much, and that is a tuning problem, not a UI problem.

Check yourself

Which of these tools can safely run without human approval?

Answer the check above to continue.

Logs you can read six weeks later

When a customer asks why their signup was held, you need to reconstruct what the agent saw, what it called and what it said. The onEvent hook from lesson 1 already emits every model turn and tool call. Write those to a file with a run id, and pseudonymise personal data on the way in:

logger.js

import { appendFileSync } from 'node:fs';
import { createHash, randomUUID } from 'node:crypto';

const SALT = process.env.LOG_SALT || 'change-me';
const PERSONAL = new Set(['email', 'ip', 'normalizedEmail']);
const pseudo = (v) => createHash('sha256').update(SALT + String(v).toLowerCase()).digest('hex').slice(0, 16);

export function makeLogger({ runId = randomUUID(), file = 'agent-events.jsonl' } = {}) {
  return (e) => {
    const input = e.input && Object.fromEntries(Object.entries(e.input).map(([k, v]) => [k, PERSONAL.has(k) ? pseudo(v) : v]));
    appendFileSync(file, `${JSON.stringify({ ts: new Date().toISOString(), runId, ...e, ...(input && { input }) })}\n`);
  };
}

Pass onEvent: makeLogger({ runId: review.id }) to triage and every line ties back to a review. A model event carries the stop reason and token usage; a tool event carries the tool, the pseudonymised input, whether it failed and how long it took. Hashing with a salt means you can still answer "how many times did we see this address?" without the log holding the address. That is pseudonymisation, not anonymisation: under GDPR the hashes are still personal data, so the log still needs a retention period. Note too that the logger only rewrites top-level input fields; the agent's free-text summary can mention an address, which is one more reason to keep the full review record in your database with proper access control rather than in log files.

From those logs, watch five numbers per week: the eval pass rate on every change, the share of runs that end in anything other than a submitted recommendation, the share with evidence problems, credits per run, and the reviewer override rate from lesson 3. Each one going the wrong way points at a different fix, and none of them needs anyone to read transcripts to notice.

You have built the whole arc of this certification: connecting agents over MCP, grounding them in live data, and now writing, testing and fencing your own. When you are ready, take the exam on the AI Agent Developer certification page.

Finished the lesson?Mark it complete to track your progress toward the certification.

Next: Exam APV-220 →

Evaluate and guard your agents before production

Make the agent testable first

Write a case set, then record the world once

What else belongs in a case set

Guard the tools, not just the prompt

Human approval for anything consequential

Logs you can read six weeks later

Run every example on your own key. Free plan, no card.

Scaling up?