Build a Signup Fraud Review Agent with Human Approval

Your signup pipeline already flags suspicious registrations: a burst from one network, an address pattern you have seen before, a datacenter IP on a consumer product. Somebody then has to look at each flag, run the same five checks, and decide. It is repetitive work, and it is exactly the kind of work people want to hand to an agent.

Hand over the investigation. Keep the decision.

This lesson builds an agent that does the legwork on a flagged signup: validates the email, checks the IP, looks for earlier accounts by the same person, and writes a short recommendation with every claim tied to a specific field from a specific tool. Then it stops. The recommendation lands in a review queue, a person makes the call, and only the person's decision changes anything. The agent has no tool that can approve, suspend or email anyone, and that absence is the most important design decision in the lesson.

There are three separate reasons, and any one of them would be enough.

The signals are probabilistic and the costs are not symmetric. A datacenter IP means the traffic came from a cloud provider. That covers scrapers and fraud rings, and it also covers someone on a corporate VPN, a developer testing from a cloud workstation and a privacy-conscious person on a commercial VPN. Wrongly rejecting a real customer costs you the customer and probably a complaint. Wrongly approving a throwaway account costs you a few credits of free tier. A model weighing those signals will sometimes weigh them wrong, and you want a person between that mistake and the customer.

Automated decisions about people carry legal weight. In the EU and UK, GDPR Article 22 restricts decisions based solely on automated processing that significantly affect a person, and refusing someone service can qualify. Other jurisdictions have similar rules. A human who genuinely reviews the evidence and can disagree changes the nature of the decision. A human who clicks "approve all" does not, which is why the review screen below shows evidence first. Take advice on your own obligations; the design here is what makes compliance possible, not a guarantee of it.

The model's write-up can be wrong in ways that look right. It may say "the IP is a VPN" when the tool said datacenter, not VPN. You will catch that in code, by checking every evidence item against the actual tool results, but a mistake that slips through should land on a reviewer's screen, not on an account.

The evidence, and what each field is worth

Three APIs from the catalog, all already in your catalog.json from lesson 1.

Email Validator is the richest source. Try it on a sub-addressed Gmail address:

Try it liveEmail Validator20 credits

GEThttps://api.apiverve.com/v1/emailvalidator

1
2

curl "https://api.apiverve.com/v1/emailvalidator?email=Jane.Doe%2Bnewsletter%40gmail.com" \
  -H "x-api-key: YOUR_API_KEY"

isSubaddressed is true and normalizedEmail is [email protected]: the dots and the +tag are gone.

normalizedEmail is the field that makes this agent useful. Gmail ignores dots and anything after a +, so [email protected] and [email protected] are the same inbox. Someone collecting repeat free trials relies on exactly that. Look up existing accounts by the normalised address and the repeat shows up immediately.

VPN Proxy Detector answers two yes-or-no questions about an IP:

Try it liveVPN Proxy Detector10 credits

GEThttps://api.apiverve.com/v1/vpndetector

1
2

curl "https://api.apiverve.com/v1/vpndetector?ip=173.172.81.20" \
  -H "x-api-key: YOUR_API_KEY"

Residential: is_vpn and is_datacenter are both false.

IP Lookup locates it:

Try it liveIP Lookup10 credits

GEThttps://api.apiverve.com/v1/iplookup

1
2

curl "https://api.apiverve.com/v1/iplookup?ip=8.8.8.8" \
  -H "x-api-key: YOUR_API_KEY"

region and regionName come back null for this address. Null means unknown, not false.

That last result is a useful warning. Some fields are null for some inputs, and city, coordinates, ASN and the EU flag are premium on IP Lookup, as is_tor and risk_level are on the VPN detector. The agent must treat a missing value as unknown. A prompt that says "reject if is_tor is true" will behave strangely on a plan where is_tor never arrives.

Here is how the signals should be weighted. This table goes into the system prompt in condensed form, and it is the part of the agent you will tune most:

Signal	Field	Weight
Throwaway email domain	`isDisposable: true`	Strong
Mail cannot be delivered	`canReceiveMail: false` or `isMxValid: false`	Strong
Same inbox already has an account	`normalizedEmail` matches an existing account	Strong for trial abuse, weak otherwise
Plus-addressing	`isSubaddressed: true`	Neutral alone; context for the match above
Cloud or hosting network	`is_datacenter: true`	Weak alone
Commercial VPN	`is_vpn: true`	Weak alone
Shared mailbox	`isRoleAccount: true`	Not a fraud signal
Country or region	`countryName`, `regionName`	Never a reason on its own

The last row deserves a sentence of its own. Rejecting signups because of where an IP is located turns a fraud check into a discrimination problem, and IP geolocation is wrong often enough that it would not be a fair proxy even if that were acceptable. Location is context for a reviewer, for example when it contradicts other information the person gave you, never a verdict.

Check yourself

A flagged signup has is_vpn true and every email check is clean: a deliverable address, not disposable, no existing account. What should the agent recommend?

Tools: three APIs, one database lookup, one way out

The agent gets the three APIVerve tools, generated from the catalog as before, plus two hand-written ones. find_accounts queries your own database by normalised email; it is read-only, and it is the only access to your data the agent has. submit_recommendation is the way out: a structured form the agent fills in, which ends the loop.

A stand-in database for this lesson:

db.js

const ACCOUNTS = [
  { id: 'acct_1001', normalizedEmail: '[email protected]', createdAt: '2026-08-02', status: 'trial_ended' },
];

export function findAccountsByNormalizedEmail(normalizedEmail) {
  return ACCOUNTS.filter((a) => a.normalizedEmail === normalizedEmail.toLowerCase());
}

And the agent:

triage.js

import { runAgent } from './agent.js';
import { apiTools, executeApiTool } from './tools.js';
import { findAccountsByNormalizedEmail } from './db.js';

const SIGNAL_APIS = ['emailvalidator', 'vpndetector', 'iplookup'];

const LOCAL_TOOLS = [
  {
    name: 'find_accounts',
    description: 'Finds existing accounts in our database with the same normalized email address (use the normalizedEmail field from emailvalidator). Returns count and the accounts\' ids, creation dates and status. Read-only. Free.',
    schema: {
      type: 'object',
      properties: { normalizedEmail: { type: 'string' } },
      required: ['normalizedEmail'], additionalProperties: false,
    },
  },
  {
    name: 'submit_recommendation',
    description: 'Submits your recommendation for the human reviewer and ends the review. Every claim in summary must be supported by an evidence item naming the tool, the exact field and the value it returned.',
    schema: {
      type: 'object',
      properties: {
        recommendation: { type: 'string', enum: ['approve', 'request_verification', 'reject'] },
        confidence: { type: 'string', enum: ['low', 'medium', 'high'] },
        summary: { type: 'string', description: 'Three sentences at most, written for the reviewer.' },
        evidence: {
          type: 'array',
          items: {
            type: 'object',
            properties: {
              tool: { type: 'string' },
              field: { type: 'string' },
              value: { type: 'string', description: 'The value exactly as the tool returned it.' },
              why: { type: 'string' },
            },
            required: ['tool', 'field', 'value', 'why'], additionalProperties: false,
          },
        },
      },
      required: ['recommendation', 'confidence', 'summary', 'evidence'], additionalProperties: false,
    },
  },
];

const SYSTEM = `You investigate signups that automated checks flagged, and write a recommendation for a human reviewer.
You cannot approve, reject or contact anyone. A person decides.
Gather evidence with the tools, then call submit_recommendation once.
Weighting:
- isDisposable true, canReceiveMail false or isMxValid false: strong evidence against.
- normalizedEmail matching an existing account: strong evidence of a repeat trial. Say how many and when.
- is_vpn or is_datacenter true: weak on their own. Many real customers use VPNs and corporate networks.
- isRoleAccount true is not a fraud signal.
- Never treat a country or region as a reason on its own.
- A null or missing field means unknown, not false.
- When evidence is mixed or thin, recommend request_verification.`;

export async function triage(signup) {
  const observed = []; // every successful result, for checking evidence later

  async function execute(name, input) {
    if (SIGNAL_APIS.includes(name)) {
      const r = await executeApiTool(name, input);
      if (!r.isError) {
        try { observed.push({ tool: name, data: JSON.parse(r.content) }); } catch { /* truncated result */ }
      }
      return r;
    }
    if (name === 'find_accounts') {
      const accounts = findAccountsByNormalizedEmail(input.normalizedEmail);
      const data = { count: accounts.length, accounts };
      observed.push({ tool: name, data });
      return { content: JSON.stringify(data), isError: false };
    }
    if (name === 'submit_recommendation') return { content: 'Submitted.', isError: false, final: input };
    return { content: `Unknown tool ${name}.`, isError: true };
  }

  // Only what the investigation needs: no name, no payment details, no free-text profile.
  const prompt = `Flagged signup ${signup.signupId}
email: ${signup.email}
ip: ${signup.ip}
flagged because: ${signup.flaggedBecause}
signed up at: ${signup.createdAt}`;

  const run = await runAgent({
    system: SYSTEM, prompt, execute, maxTurns: 6,
    tools: [...apiTools(SIGNAL_APIS), ...LOCAL_TOOLS],
  });

  const recommendation = run.stopReason === 'final' ? run.final : null;
  const problems = recommendation ? checkEvidence(recommendation, observed) : [`Agent stopped without a recommendation (${run.stopReason}).`];
  return { recommendation, problems, observed };
}

export function checkEvidence(rec, observed) {
  const problems = [];
  if (rec.evidence.length === 0) problems.push('No evidence given.');
  for (const e of rec.evidence) {
    const hit = observed.find((o) => o.tool === e.tool && Object.hasOwn(o.data, e.field));
    if (!hit) problems.push(`No ${e.tool} result has a field "${e.field}".`);
    else if (String(hit.data[e.field]) !== String(e.value)) {
      problems.push(`${e.tool}.${e.field} was ${JSON.stringify(hit.data[e.field])}, not ${JSON.stringify(e.value)}.`);
    }
  }
  return problems;
}

Walk through what this enforces.

The prompt contains only what the investigation needs. Email, IP, the reason for the flag and a timestamp. Not the person's name, not their payment details, not anything they typed into a profile. Everything in the prompt goes to your model provider, and much of it may end up in your logs. Data minimisation is a GDPR principle, and it is also simply less to leak. It has a security benefit too: a free-text field written by the person you are screening is an injection channel, and leaving it out closes the channel.

observed records exactly what the tools returned, and checkEvidence compares each evidence item with it. If the agent writes vpndetector / is_vpn / true and the tool actually said false, the review carries a visible problem. The value field is a string in the schema so that the comparison is uniform; String(true) is "true", which is what the model will write.

A failed run still produces a review. If the agent stops early, hits the turn limit or a tool fails fatally, recommendation is null and the problems list says why. The flagged signup still goes into the queue with whatever evidence was collected. An agent outage should degrade to "a human reviews it unaided", which is what happened before the agent existed, never to "the signup is silently approved".

Check yourself

The agent's summary says the IP 'belongs to a VPN provider', and its evidence item reads tool vpndetector, field is_vpn, value true. The recorded result has is_vpn false and is_datacenter true. What does checkEvidence do, and why does it matter?

Answer the check above to continue.

The review queue: where the decision actually happens

Install Express 4 and put the agent behind three routes: one your signup pipeline calls when it flags someone, one that lists pending reviews, and one where a reviewer records a decision.

npm install express@4

server.js

import express from 'express';
import { randomUUID } from 'node:crypto';
import { triage } from './triage.js';

const app = express();
app.use(express.json());
const reviews = new Map(); // use your database in production

// Called by your signup pipeline. Put all routes behind your admin authentication.
app.post('/flagged', async (req, res) => {
  const { signupId, email, ip, flaggedBecause } = req.body;
  if (!signupId || !email || !ip) return res.status(400).json({ error: 'signupId, email and ip are required' });
  const result = await triage({ signupId, email, ip, flaggedBecause, createdAt: new Date().toISOString() });
  const id = randomUUID();
  reviews.set(id, { id, signupId, status: 'pending', ...result });
  res.status(202).json({ id });
});

app.get('/reviews', (req, res) => {
  res.json([...reviews.values()].filter((r) => r.status === 'pending'));
});

app.post('/reviews/:id/decision', async (req, res) => {
  const review = reviews.get(req.params.id);
  if (!review || review.status !== 'pending') return res.status(404).json({ error: 'No pending review with that id' });
  const { decision, reviewer, note } = req.body;
  if (!['approve', 'request_verification', 'reject'].includes(decision) || !reviewer) {
    return res.status(400).json({ error: 'decision and reviewer are required' });
  }
  Object.assign(review, {
    status: 'decided', decision, reviewer, note,
    decidedAt: new Date().toISOString(),
    agreedWithAgent: review.recommendation ? decision === review.recommendation.recommendation : null,
  });
  await applyDecision(review.signupId, decision);
  res.json(review);
});

async function applyDecision(signupId, decision) {
  // Your existing account code: unlock, require verification, or close.
  console.log(`Applying ${decision} to ${signupId}`);
}

app.listen(3000, () => console.log('Review queue on http://localhost:3000'));

Try it with a repeat-trial pattern. The address below normalises to [email protected], which already has an account in db.js:

node server.js
curl -s -X POST localhost:3000/flagged -H 'Content-Type: application/json' \
  -d '{"signupId":"su_2002","email":"[email protected]","ip":"103.62.49.210","flaggedBecause":"datacenter IP"}'
curl -s localhost:3000/reviews

A sound recommendation here is reject or request_verification, citing isSubaddressed, normalizedEmail, find_accounts.count of 1 and is_datacenter, and saying in its summary that the datacenter IP is supporting rather than decisive. If it recommends reject on the datacenter IP alone, your weighting instructions need work. Lesson 4 shows how to catch that systematically instead of by eye.

Three properties of this queue matter more than its simplicity suggests.

Only applyDecision changes anything, and only a person reaches it. The agent's code path ends at reviews.set. There is no route, flag or timeout that turns a recommendation into a decision. Some teams add "auto-approve when the agent recommends approve with high confidence" a month in, because the queue is long. If you do that, you have built an automated decision system, and every argument at the top of this lesson applies to it.

The reviewer is recorded. reviewer and decidedAt are the audit trail. When a customer asks why their account was closed, "the agent decided" is not an answer you want to give.

Agreement is measured. agreedWithAgent costs one line and gives you the single most useful number for this agent: how often people overrule it. A high override rate means the agent needs work. An override rate near zero can mean the agent is excellent, or that reviewers have stopped reading. Look at how long reviews take as well: decisions made in two seconds are not reviews.

Keep retention in mind as well. A review stores an email address, an IP address and a written assessment of a person. Decide how long you keep decided reviews, who can read them, and how they are deleted when the account is, the same way you would for any other record about a user.

You now have two agents built on one loop. Before either goes near production traffic, you need to know how often they get things right and what stops them when they do not: Evaluate and guard your agents.

Finished the lesson?Mark it complete to track your progress toward the certification.

Next: Evaluate and guard your agents before production →

Build a signup triage agent that recommends, never acts

The evidence, and what each field is worth

Tools: three APIs, one database lookup, one way out

The review queue: where the decision actually happens

Run every example on your own key. Free plan, no card.

Scaling up?