A crawler that works is easy. A crawler that is still welcome six months later takes a little more thought. The site on the other end pays for every request you make: bandwidth, CPU, sometimes a per-request bill from their own host. When a crawler hits a small site twenty times a second, the owner notices, blocks the IP range, and in the worst cases sends a letter. Most of the scrapers that get blocked are not malicious. They are careless.
The APIs in this course do the fetching for you. When you send a URL to Website to Text or URL Title, the request to the target site is made from APIVerve's infrastructure, not from your server. That removes the browser, the proxy and the HTML parser from your code. It does not remove the decision about which URLs to send and how often. That decision is yours, and so are its consequences. This lesson builds the three pieces that encode it: a robots.txt check, a queue that paces each host, and a cache that stops you asking for the same page twice.
What you are allowed to fetch
Three sources of rules apply to any page, in rough order of how often they bite.
robots.txt is the site owner's machine-readable request: which paths crawlers should leave alone, sometimes how long to wait between requests. It was standardised as RFC 9309 in 2022 after nearly thirty years as a convention. It is not access control, and nothing technically stops you ignoring it. But it is the clearest statement of intent you will get, operators treat violations as hostile, and in a dispute it is the first thing anyone will ask about.
Terms of service are a contract question. A site's terms may forbid automated access or commercial reuse of its content. How enforceable that is depends on the jurisdiction and on whether you ever agreed to them: an account you signed up for is a much stronger agreement than a footer link you never clicked. The long-running US case hiQ Labs v. LinkedIn is the usual reference. The courts found that scraping public pages was not "access without authorization" under the federal anti-hacking statute, and hiQ still lost later on breach of LinkedIn's user agreement. Public is not the same as permitted. If a page needs a login, treat it as off-limits to your pipeline unless you have a written agreement.
Personal data law applies to what you keep, not just what you fetch. Under GDPR and laws modelled on it, a name, email address or profile photo on a public page is still personal data, and collecting it is processing that needs a lawful basis, a purpose and a retention limit. The practical rule for pipelines: do not store personal data you have no use for. If you are building a price tracker, the reviewer names on the product page are a liability, not an asset. Drop them at extraction time, before they reach a database you will have to audit later.
None of this is legal advice, and the edges are genuinely unsettled. But a pipeline that respects robots.txt, stays out of logged-in areas, paces itself and keeps only what it needs is on the right side of nearly every dispute that has actually happened.
Your pipeline extracts product prices from public retailer pages. The pages also show reviewer names and avatars. What should happen to the reviewer data?
Answer the check above to continue.
Pace each host, not just the whole crawler
There are two speed limits in play and they protect different people.
The first is yours to APIVerve. Your plan has a per-minute rate limit and a separate ceiling on concurrent calls; exceed either and you get a 429 with a retry-after header. That limit protects the platform and your credit balance.
The second is the one you owe each target site, and nothing enforces it but your code. Ten URLs on ten different hosts can run in parallel without bothering anyone. Ten URLs on one host, fired at once, is a burst that small sites feel. So the queue needs two knobs: a global concurrency cap, and a per-host gap between requests that respects Crawl-delay when the site sets one.
A queue that does both is about thirty lines. No dependency is needed.
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
export function createQueue({ concurrency = 3 } = {}) {
let active = 0;
const waiting = [];
const nextSlot = new Map(); // host -> earliest time the next request may start
async function acquire() {
if (active < concurrency) {
active++;
return;
}
await new Promise((resolve) => waiting.push(resolve)); // slot is handed over by release()
}
function release() {
const next = waiting.shift();
if (next) next();
else active--;
}
async function run(host, gapMs, task) {
await acquire();
try {
const now = Date.now();
const at = Math.max(now, nextSlot.get(host) ?? 0);
nextSlot.set(host, at + gapMs); // reserve before awaiting, so parallel callers queue up behind it
if (at > now) await sleep(at - now);
return await task();
} finally {
release();
}
}
return { run };
}Two details matter. release() hands its slot directly to the next waiter instead of decrementing and letting the waiter re-increment; the naive version briefly lets one extra task in under load. And nextSlot is updated before the await, so three tasks for the same host reserve consecutive slots rather than all reading the same timestamp and firing together.
A sleeping task holds a concurrency slot while it waits for its host. With three slots and a crawl dominated by one host, that means the crawl runs at that host's pace, which is exactly the point. If you mix many hosts, sort or interleave the URL list by host so slots are not all parked on the same slow site.
Cache before you queue
The cheapest polite request is the one you never make. A page title that was fetched an hour ago is almost certainly the same now. Caching also matters for your own bill: URL Title costs 10 credits a call and Website to Text costs 25, and a crawler restarted after a crash will happily re-pay for everything it already did.
curl -X POST "https://api.apiverve.com/v1/urltitle" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com/"}'title is the page's <title> tag. The h1 array is filled on paid plans only.
Put the cache in front of the queue, not inside it. A cache hit should cost neither credits nor a host slot.
import { createHash } from 'node:crypto';
import { mkdir, readFile, writeFile } from 'node:fs/promises';
const DIR = './cache';
export async function cached(api, body, ttlMs, fetcher) {
const key = createHash('sha256').update(api + JSON.stringify(body)).digest('hex');
const file = `${DIR}/${key}.json`;
try {
const hit = JSON.parse(await readFile(file, 'utf8'));
if (Date.now() - hit.at < ttlMs) return hit.data;
} catch {
// no entry yet, or unreadable: fall through and fetch
}
const data = await fetcher();
await mkdir(DIR, { recursive: true });
await writeFile(file, JSON.stringify({ at: Date.now(), data }));
return data;
}Choose the TTL per API, by how fast the answer changes: a day for titles, an hour for news article text, minutes for a page you are monitoring. Never cache errors. A timeout this minute says nothing about the next.
Put it together
The client below is the minimal one from the Fundamentals courses, kept short: POST JSON, throw on a non-ok envelope.
export async function verve(api, body) {
const res = await fetch(`https://api.apiverve.com/v1/${api}`, {
method: 'POST',
headers: { 'x-api-key': process.env.APIVERVE_API_KEY, 'Content-Type': 'application/json' },
body: JSON.stringify(body),
signal: AbortSignal.timeout(30000),
});
const json = await res.json().catch(() => null);
if (!res.ok || json?.status !== 'ok') throw new Error(json?.error || `HTTP ${res.status}`);
return json.data;
}And the crawler itself: robots check first, then cache, then the paced queue.
import { robotsFor, isAllowed } from './robots.js';
import { createQueue } from './queue.js';
import { cached } from './cache.js';
import { verve } from './verve.js';
const MIN_GAP_MS = 2000;
const DAY = 24 * 60 * 60 * 1000;
const queue = createQueue({ concurrency: 3 });
async function titleFor(url) {
const robots = await robotsFor(url);
if (!isAllowed(robots, url)) return { url, skipped: 'robots.txt' };
const host = new URL(url).host;
const gap = Math.max(MIN_GAP_MS, robots.delay * 1000);
try {
const data = await cached('urltitle', { url }, DAY, () =>
queue.run(host, gap, () => verve('urltitle', { url })),
);
return { url, title: data.title };
} catch (err) {
return { url, error: err.message };
}
}
const urls = process.argv.slice(2);
console.table(await Promise.all(urls.map(titleFor)));export APIVERVE_API_KEY=YOUR_API_KEY
node crawl.js https://example.com/ https://www.iana.org/help/example-domains https://www.iana.org/domainsRun it twice. The second run returns instantly and spends nothing, because every title comes from ./cache. The two IANA URLs share a host, so the second waits at least two seconds behind the first even though a concurrency slot was free.
Swapping in full-text extraction is one line: change 'urltitle' to 'websitetotext' and read data.text. Everything else, including the politeness, stays the same.
curl -X POST "https://api.apiverve.com/v1/websitetotext" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com/"}'text is the body copy. language is the page's declared language, and date or publisher are null when the page does not mark them up.
The checklist
Before a crawler runs unattended, it should be able to answer yes to each of these:
| Question | Where it lives |
|---|---|
| Does it read and obey robots.txt, including 5xx-means-stop? | robots.js |
| Does it name itself and give a contact? | USER_AGENT |
Is there a per-host gap, and does it honour Crawl-delay? | queue.js, crawl.js |
| Is global concurrency capped below your plan's ceiling? | createQueue({ concurrency }) |
| Are repeat fetches served from a cache with a sensible TTL? | cache.js |
| Does it skip logged-in areas and drop personal data it has no use for? | Your URL list and your extraction code |
The next lesson takes the raw text that comes back and turns it into something you can search and store: Clean and structure extracted text.