Here is an experiment worth running before you read anything else. Send the string Googlebot/2.1 (+http://www.google.com/bot.html) to a bot detector twice: once with the address 66.249.66.1, once with 185.220.101.1. Both calls report a known, trusted search engine crawler. Only one of them is Google. The first comes from Google's published crawler range and the detector marks it verified: true with a risk score of 0. The second comes from the Tor exit you met in lessons 2 and 3, and the same user agent earns verified: false, a risk score of 70 and a high risk level.
That is the whole lesson in miniature. A user agent is text the client chooses to send. Honest software uses it to describe itself; dishonest software uses it to describe whatever gets it through your door. You will learn to read it for what it is worth, catch the automation that does not bother to hide, verify the crawlers that claim to be someone important, and then put every signal from this course into one scoring function.
A user agent is a claim
The User-Agent header arrives with every HTTP request. Browsers fill it in; so do curl, Python's requests, headless Chrome, search engine crawlers and every scraping framework. Changing it is a single line of configuration in all of them, which gives you two populations to think about:
- Clients that tell the truth. Real browsers, well-behaved crawlers, monitoring tools, libraries left on their defaults. For these the user agent is accurate and genuinely useful: analytics, compatibility decisions, spotting a script that someone forgot to disguise.
- Clients that lie. Scrapers and fraud tools that copy a current Chrome string, or claim to be Googlebot because some sites wave Googlebot through.
So user agent analysis has a strange property: it is excellent at catching careless automation and nearly useless against careful automation. That is still worth a lot, since most abuse is careless. But it means a browser-looking user agent proves nothing, and a decision that rests on the user agent alone is one that a determined attacker controls.
Parse what the client claims
User Agent Parser (useragentparser) turns the string into structured fields. It is a POST with a ua field and costs 2 credits, the cheapest call in this course.
curl -X POST "https://api.apiverve.com/v1/useragentparser" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"ua":"Mozilla/5.0 (iPhone; CPU iPhone OS 17_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Mobile/15E148 Safari/604.1"}'browser.name is Mobile Safari, os is iOS 17.4, deviceType is mobile.
On every plan you get browser.name and browser.version, os.name and os.version, the device object (vendor, model, type when known), cpu, isMobile, isBot and deviceType, which is one of desktop, mobile, tablet, console, smarttv, wearable, embedded, xr or bot. Premium plans add browser.major, the engine name and version, and a human-readable summary such as Chrome 120 on Windows 10.
Two things in modern browser strings will catch you out if you read them literally.
Windows 11 says it is Windows 10. Browsers on Windows 11 still send Windows NT 10.0, so the parser reports os.version: "10" for both. There is no way to tell them apart from the user agent; browsers that support client hints expose the real platform version in the Sec-CH-UA-Platform-Version header if you request it.
Chrome versions end in zeros. Chrome reduced the detail in its user agent, so a desktop Chrome string carries Chrome/120.0.0.0: the major version is real and the rest is frozen. The parser reports browser.version: "120.0.0.0". Do not build logic on minor or build numbers you will never see.
The parser is built for browser strings. Give it something that is not a browser, such as python-requests/2.31.0, and it returns empty browser and os objects and isBot: false, because it is not designed to identify HTTP libraries. That is not a failure of the parser so much as a sign you asked the wrong tool. Bot questions go to the bot detector.
Your analytics shows zero Windows 11 users and many Windows 10 users, all from parsed user agents. What is the most likely explanation?
Answer the check above to continue.
Combine every signal into one decision
You now have four families of signal: what the client claims (user agent), whether the network hides it (VPN, datacenter, Tor), whether it has a bad record (blacklist), and whether it is automated (bot detection). Individually each one is noisy. Together they are much harder to fake, because an attacker has to get every one of them right at once.
The scoring function below adds points per signal and maps the total to the same four actions you used in lesson 2. It reuses networkSignal() from lesson 2 and the verve() helper from lesson 1, and it follows three rules:
- Cheapest and most cacheable first. User agent strings repeat enormously: a few hundred distinct strings cover most of your traffic. Cached by string, the bot check is nearly free after the first hour.
- Stop early when the answer is already clear. A verified Googlebot needs nothing else; a score already past the block threshold does not need another lookup.
- Do not count one fact twice. A Tor exit is also a datacenter and is usually blacklisted. Score the strongest version of the fact once.
import { verve } from './verve.js';
import { networkSignal } from './network.js';
const UA_TTL_MS = 6 * 60 * 60 * 1000;
const uaCache = new Map();
async function botSignal(ua, ip) {
const claimsCrawler = /googlebot|bingbot|gptbot|applebot/i.test(ua);
const key = claimsCrawler ? `${ua}|${ip}` : ua; // verification depends on the IP
const hit = uaCache.get(key);
if (hit && hit.expires > Date.now()) return hit.value;
const params = claimsCrawler ? { ua, ip } : { ua };
const d = await verve('botdetector', params);
const value = {
isBot: d.isBot,
reputation: d.bot?.reputation ?? null, // premium
verified: d.verified ?? null, // premium; null when nothing to verify
claimsCrawler,
};
uaCache.set(key, { value, expires: Date.now() + UA_TTL_MS });
return value;
}
async function blacklisted(ip) {
const d = await verve('ipblacklistlookup', { ip });
return d.isIPBlacklisted;
}
export async function assess({ ip, ua, route }) {
const reasons = [];
let score = 0;
const add = (points, reason) => { score += points; reasons.push(reason); };
if (!ua) add(30, 'no-user-agent');
else {
try {
const bot = await botSignal(ua, ip);
if (bot.claimsCrawler && bot.verified === true) return { score: 0, action: 'allow', reasons: ['verified-crawler'] };
if (bot.claimsCrawler && bot.verified === false) add(50, 'fake-crawler');
else if (bot.isBot && route !== 'api') add(30, 'automated-client');
if (bot.reputation === 'malicious') add(30, 'malicious-bot');
} catch {
reasons.push('bot-check-failed');
}
}
const net = await networkSignal(ip);
if (net.tor) add(40, 'tor');
else if (net.vpn) add(15, 'vpn');
else if (net.datacenter && route !== 'api') add(20, 'datacenter');
if (score < 70 && !net.tor) { // a Tor exit is almost always listed; don't count it twice
try {
if (net.kind !== 'private' && (await blacklisted(net.ip))) add(25, 'blacklisted');
} catch {
reasons.push('blacklist-check-failed');
}
}
const action = score >= 90 ? 'block' : score >= 50 ? 'step_up' : score >= 25 ? 'limit' : 'allow';
return { score, action, reasons };
}Wire it into the routes that matter, and log every result with its reasons:
import express from 'express';
import { assess } from './risk.js';
const app = express();
app.set('trust proxy', Number(process.env.TRUSTED_PROXIES ?? 0));
app.use(express.json());
app.post('/signup', async (req, res) => {
const risk = await assess({ ip: req.ip, ua: req.get('user-agent') ?? '', route: 'signup' });
console.log(JSON.stringify({ route: 'signup', ip: req.ip, ...risk }));
if (risk.action === 'block') return res.status(403).json({ error: 'Signup is not available from this client.' });
if (risk.action === 'step_up') return res.json({ next: 'captcha' });
res.json({ next: 'create-account', trialCredits: risk.action === 'limit' ? 'reduced' : 'standard' });
});
app.listen(3000, () => console.log('listening on :3000'));The weights are a starting point, not a finding. A fake crawler claim (50) alone triggers a step-up; Tor (40) plus a scripted user agent (30) does too; a VPN alone (15) is allowed. Run it in log-only mode first, as in lesson 2, and move the thresholds only when outcome data says a combination is genuinely worse than average on that route.
Two more signals cost nothing and belong in the same function once you have them: a missing Accept-Language header, which mainstream browsers send by default, and request rate per address or per account from your own rate limiter. The signals you already own are the hardest for an attacker to see, which makes them some of the most valuable.
Things not to do
- Do not block good bots on public pages. Search engine and link-preview crawlers are how people find and share you. Control them with
robots.txtand verified-crawler logic, not a blanketisBotblock. - Do not trust
isBot: false. It means the client did not identify itself as automated. A careful scraper looks exactly like Chrome on Windows. - Do not fingerprint more than you need. A user agent combined with an IP and other headers can single out an individual, which brings it under the same privacy rules as the IP in lesson 1. Keep the derived verdict and reasons; drop the raw combination when your security purpose no longer needs it.
- Do not tell attackers your thresholds. Error messages say what the user can do, never which signal fired.
You have finished the network half of the verification certification. The final course moves from who is connecting to who is paying: card and bank details, tax IDs and sanctions screening. Continue to Payments and compliance.