Chain APIs in Node.js: Scrape, Summarize and Score Sentiment

The previous projects each called one or two APIs and used the answers directly. Real features are often a chain: one call produces something, the next call consumes it, and the thing you show the user only exists at the end. The code that connects those calls is where most of the engineering lives. Which steps depend on which? What happens when the third step fails but the first two worked? How much did the whole run cost?

You will build a command-line pipeline that takes one or more URLs and, for each, extracts the page's text, summarises it, and scores its tone. It is the core of a dozen real tools: a reading-list digest, a competitor-news monitor, a review triage queue. The three APIs are almost incidental. The structure is the lesson.

The three steps and their contracts

Every link in a chain has a contract: what goes in, what comes out, and what the next step needs from it. Write those down before writing code, because the mismatches between them are where pipelines break.

Step 1: Website to Text (websitetotext). A POST with one required body field, url. It returns text (the page's body copy with the HTML stripped), plus title, description, date, language, publisher and url. The catalog lists it at 25 credits.

Try it liveWebsite to Text25 credits

POSThttps://api.apiverve.com/v1/websitetotext

curl -X POST "https://api.apiverve.com/v1/websitetotext" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://en.wikipedia.org/wiki/QR_code"}'

url

text is long: tens of thousands of characters for a full article. That matters for the next step.

When we ran that call, text was over 76,000 characters, and the extracted title and date were not the article's title and date. Metadata extraction is a best guess on pages that do not mark it up cleanly. The body text is the dependable output; treat the rest as optional and fall back to the URL when a title looks wrong or is missing.

Step 2: Text Summarizer (textsummarizer). A POST taking text (up to 5,000 characters) and an optional sentences count, default 3. It returns summary. The word counts and reduction percentage (originalWords, summaryWords, percentDifference) are premium. 20 credits.

Try it liveText Summarizer20 credits

POSThttps://api.apiverve.com/v1/textsummarizer

curl -X POST "https://api.apiverve.com/v1/textsummarizer" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text":"A QR code is a two-dimensional matrix barcode invented in 1994 by Masahiro Hara of Denso Wave for labelling automobile parts. It is read by cameras and decoded using Reed-Solomon error correction. QR codes store data in four encoding modes: numeric, alphanumeric, byte and kanji. They are used to open web pages, add contacts, join wireless networks and make payments.","sentences":2}'

textsentences

The summary is extractive: it picks existing sentences rather than writing new ones.

The summary is extractive. It selects the sentences that carry the most weight and returns them as written, which means it never invents a claim the page did not make. It also means a sentence that starts with "It" may lose its subject. Worth knowing before you show it to users.

Step 3: Sentiment Analysis (sentimentanalysis). A POST taking text. It returns sentiment (a summed word score), comparative (that score divided by the number of words), sentimentText (a label), and the booleans isPositive and isNegative; normalizedScore is premium. 10 credits.

Try it liveSentiment Analysis10 credits

POSThttps://api.apiverve.com/v1/sentimentanalysis

curl -X POST "https://api.apiverve.com/v1/sentimentanalysis" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text":"The package arrived late and the box was broken."}'

text

Compare sentimentText with isNegative and the numeric score.

Run that one and look closely. When we called it, sentiment was -1, isNegative was true, and sentimentText was neutral. The label is a band over the score, and a mildly negative sentence falls inside the neutral band. Neither field is wrong; they answer different questions. If your feature sorts by tone, sort by comparative, which is comparable across texts of different lengths. Use the label only for display.

The cost of one URL through the whole chain is the sum of its steps: 25 + 20 + 10 = 55 credits. Calls that fail are not billed, so a run where step 3 errors costs 45. You will make the pipeline report that number itself, rather than leaving it as arithmetic you do once and forget.

The shape of the pipeline

Draw the dependencies before you write anything:

1
2

url ──► websitetotext ──► excerpt ─┬─► textsummarizer ──► summary
                                   └─► sentimentanalysis ──► tone

Step 1 must finish before anything else can start. Steps 2 and 3 both need the excerpt, but neither needs the other, so they run at the same time. That turns the pipeline's latency from the sum of three calls into the first call plus the slower of the other two.

The excerpt step in the middle is not an API. It is your code adapting one contract to the next: the summariser takes at most 5,000 characters, and a full article can be fifteen times that. You cut the text at a sentence boundary under the limit and strip noise like citation markers. Every real pipeline has glue like this, and it deserves the same care as the calls.

Failures follow the same picture. If step 1 fails there is nothing to analyse, so that URL fails as a whole. If step 2 or 3 fails, the other's result is still useful, so you return a partial result with a clear marker rather than throwing away work you paid for.

Check yourself

Step 1 succeeds, then the summarizer times out while sentiment analysis succeeds. What should the pipeline return for that URL?

Step 1: a client that knows what to retry

Set up an ES module project. No dependencies are needed; Node 18's built-in fetch does everything.

mkdir pipeline && cd pipeline
npm init -y
npm pkg set type=module
export APIVERVE_API_KEY=YOUR_API_KEY

The client is the same as in earlier projects, with one addition: it marks which failures are worth a retry. A timeout, a 5xx, or a 429 from the per-minute rate limit can succeed on a second attempt. A 400 (bad input), a 401 (bad key) or a 429 with x-api-remaining-credits at 0 (the month is spent) will fail identically every time, and retrying them only adds delay.

verve.js

const BASE = 'https://api.apiverve.com/v1';

export class VerveError extends Error {
  constructor(message, status, retryable) {
    super(message);
    this.status = status;
    this.retryable = retryable;
  }
}

export async function verve(api, body, { timeoutMs = 15000 } = {}) {
  const key = process.env.APIVERVE_API_KEY;
  if (!key) throw new Error('APIVERVE_API_KEY is not set');

  let res;
  try {
    res = await fetch(`${BASE}/${api}`, {
      method: 'POST',
      headers: { 'x-api-key': key, 'Content-Type': 'application/json' },
      body: JSON.stringify(body),
      signal: AbortSignal.timeout(timeoutMs),
    });
  } catch (err) {
    throw new VerveError(err.name === 'TimeoutError' ? 'timed out' : err.message, 0, true);
  }

  const json = await res.json().catch(() => null);
  if (!res.ok || json?.status !== 'ok') {
    const outOfCredits = res.status === 429 && res.headers.get('x-api-remaining-credits') === '0';
    const retryable = !outOfCredits && (res.status === 429 || res.status >= 500);
    throw new VerveError(outOfCredits ? 'out of credits' : json?.error || `HTTP ${res.status}`, res.status, retryable);
  }
  return json.data;
}

Step 2: steps, a ledger, and the pipeline

Each call goes through a step wrapper that does three things: retries once if the failure is retryable, times the call, and writes a line to a ledger. The ledger is the pipeline's memory. At the end it tells you which steps ran, which failed, how long each took and what the run cost, using the catalog's per-call credit figures and charging nothing for failures.

pipeline.js

import { verve } from './verve.js';

// Per-call credits from each API's catalog page.
const CREDITS = { websitetotext: 25, textsummarizer: 20, sentimentanalysis: 10 };
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function step(api, body, ledger, timeoutMs) {
  const started = Date.now();
  for (let attempt = 1; ; attempt++) {
    try {
      const data = await verve(api, body, { timeoutMs });
      ledger.push({ api, ok: true, attempt, ms: Date.now() - started, credits: CREDITS[api] });
      return data;
    } catch (err) {
      if (err.retryable && attempt < 2) {
        await sleep(1000);
        continue;
      }
      ledger.push({ api, ok: false, attempt, ms: Date.now() - started, credits: 0, error: err.message });
      throw err;
    }
  }
}

// Glue between step 1 and steps 2-3: clean, then cut under the summarizer's limit at a sentence end.
export function excerpt(text, max = 5000) {
  const clean = String(text).replace(/\[\d+\]/g, '').replace(/\s+/g, ' ').trim();
  if (clean.length <= max) return clean;
  const cut = clean.slice(0, max);
  const end = cut.lastIndexOf('. ');
  return end > max * 0.6 ? cut.slice(0, end + 1) : cut;
}

export async function analyse(url) {
  const ledger = [];
  const total = () => ledger.reduce((n, s) => n + s.credits, 0);

  let page;
  try {
    page = await step('websitetotext', { url }, ledger, 30000);
  } catch (err) {
    return { url, error: `could not read page: ${err.message}`, credits: total(), steps: ledger };
  }

  const text = excerpt(page.text ?? '');
  if (text.split(' ').length < 40) {
    return { url, error: 'page has too little text to analyse', credits: total(), steps: ledger };
  }

  const [summary, tone] = await Promise.allSettled([
    step('textsummarizer', { text, sentences: 3 }, ledger, 20000),
    step('sentimentanalysis', { text }, ledger, 15000),
  ]);

  return {
    url,
    title: page.title?.trim() || url,
    language: page.language ?? null,
    summary: summary.status === 'fulfilled' ? summary.value.summary : null,
    tone: tone.status === 'fulfilled'
      ? { perWord: tone.value.comparative, score: tone.value.sentiment, label: tone.value.sentimentText }
      : null,
    credits: total(),
    steps: ledger,
  };
}

Promise.allSettled rather than Promise.all is the key choice here. Promise.all rejects as soon as either promise rejects, which would throw away a perfectly good sentiment result because the summary failed. allSettled waits for both and tells you how each one ended, which is exactly the partial-result behaviour you designed above.

Notice also that analyse never throws. Every outcome, including total failure, comes back as an object with a credits figure. When you run fifty URLs, one bad link should produce one error line in the report, not abort the other forty-nine.

Step 3: run many URLs with a concurrency limit

The command-line runner takes URLs as arguments and pushes them through analyse a few at a time. An unbounded Promise.all over a hundred URLs would start three hundred API calls within seconds and run straight into your plan's concurrency and rate limits. A pool of three keeps at most three URLs, and at most six calls, in flight.

run.js

import { analyse } from './pipeline.js';

async function pool(items, size, fn) {
  const out = [];
  for (let i = 0; i < items.length; i += size) {
    out.push(...(await Promise.all(items.slice(i, i + size).map(fn))));
  }
  return out;
}

const urls = process.argv.slice(2);
if (!urls.length) {
  console.error('Usage: node run.js <url> [url...]');
  process.exit(1);
}

const results = await pool(urls, 3, analyse);

for (const r of results) {
  console.log(`\n${r.title ?? r.url}`);
  if (r.error) {
    console.log(`  failed: ${r.error}`);
  } else {
    console.log(`  summary: ${r.summary ?? '(summary step failed)'}`);
    console.log(r.tone
      ? `  tone: ${r.tone.label} (score ${r.tone.score}, ${r.tone.perWord.toFixed(3)} per word)`
      : '  tone: (sentiment step failed)');
  }
  console.log(`  steps: ${r.steps.map((s) => `${s.api} ${s.ok ? 'ok' : 'FAILED'} ${s.ms}ms`).join(', ')}`);
  console.log(`  credits: ${r.credits}`);
}

console.log(`\nTotal credits: ${results.reduce((n, r) => n + r.credits, 0)} for ${urls.length} URL(s)`);

Try it on a couple of real pages:

node run.js https://en.wikipedia.org/wiki/QR_code https://en.wikipedia.org/wiki/Exchange_rate

Each URL that completes reports 55 credits. Try a URL that does not exist and watch the ledger show step 1 failing, the steps that never ran, and a cost of zero.

Here is the same analyse in Python, with a thread pool standing in for Promise.allSettled. Each future is resolved separately, so one step's failure does not hide the other's result.

pipeline.py

import os, re, sys
from concurrent.futures import ThreadPoolExecutor
import requests

BASE = "https://api.apiverve.com/v1"
CREDITS = {"websitetotext": 25, "textsummarizer": 20, "sentimentanalysis": 10}


def verve(api, body, timeout):
    res = requests.post(f"{BASE}/{api}", json=body,
                        headers={"x-api-key": os.environ["APIVERVE_API_KEY"]}, timeout=timeout)
    data = res.json()
    if res.status_code != 200 or data.get("status") != "ok":
        raise RuntimeError(data.get("error") or f"HTTP {res.status_code}")
    return data["data"]


def excerpt(text, limit=5000):
    clean = re.sub(r"\s+", " ", re.sub(r"\[\d+\]", "", text or "")).strip()
    if len(clean) <= limit:
        return clean
    cut = clean[:limit]
    end = cut.rfind(". ")
    return cut[: end + 1] if end > limit * 0.6 else cut


def analyse(url):
    credits = 0
    try:
        page = verve("websitetotext", {"url": url}, 30)
        credits += CREDITS["websitetotext"]
    except Exception as e:
        return {"url": url, "error": f"could not read page: {e}", "credits": credits}

    text = excerpt(page.get("text"))
    with ThreadPoolExecutor(max_workers=2) as ex:
        fut_sum = ex.submit(verve, "textsummarizer", {"text": text, "sentences": 3}, 20)
        fut_tone = ex.submit(verve, "sentimentanalysis", {"text": text}, 15)
        result = {"url": url, "title": (page.get("title") or "").strip() or url}
        for name, fut in (("textsummarizer", fut_sum), ("sentimentanalysis", fut_tone)):
            try:
                result[name] = fut.result()
                credits += CREDITS[name]
            except Exception as e:
                result[name] = None
                result[f"{name}_error"] = str(e)
    result["credits"] = credits
    return result


if __name__ == "__main__":
    for u in sys.argv[1:]:
        r = analyse(u)
        print(r["url"], "->", r.get("error") or (r["textsummarizer"] or {}).get("summary"))
        print("   tone:", (r.get("sentimentanalysis") or {}).get("comparative"), "| credits:", r["credits"])

Check yourself

You need to analyse 200 URLs. Why use a pool of 3 instead of Promise.all over all 200?

Answer the check above to continue.

Harden it

Cache the expensive step. Step 1 is the most expensive call, and pages you have already read rarely change within the hour. Cache page.text by URL; a re-run then costs 30 credits per URL instead of 55, and cached URLs also skip the slowest call.

Consider a cheaper first step. URL to Markdown is listed at 10 credits rather than 25 and returns markdown and wordCount. Markdown carries heading and list syntax, so you would strip that in the excerpt step. It is a real trade-off worth measuring on your own pages, not a free swap.

Budget before you run. You know the maximum cost before any call: 55 credits times the number of URLs. Print it and ask for confirmation above a threshold. A pipeline that can spend a few thousand credits in one command should say so first.

Stop on an empty balance. When a step fails with "out of credits", every later call will fail the same way. Check for it in the runner and stop the pool rather than collecting two hundred identical errors.

Keep the ledger. Writing each run's ledger to a file gives you the latency and failure rate of every step over time, which is how you find out which link in a chain is the weak one.

For the full set of retryable and non-retryable errors, see error handling. For limits on parallel calls per plan, see making requests.

Ideas to extend it

Feed in a feed. Put RSS to JSON in front of step 1 to turn a news feed into the URL list, and you have a daily digest.
Add a language gate. websitetotext returns language. Skip the summary and sentiment steps for languages your audience does not read, and save 30 credits per page.
Rank by tone. Sort a batch of product reviews by comparative to put the most negative at the top of a support queue.
More workflows. The recipes collection has more multi-API chains to take apart and rebuild.

That completes the five projects. You have built a signup gate, a weather widget, a currency converter, a QR check-in system and a multi-step pipeline, and each one taught a habit that carries over: decide what failure means, keep the key on the server, cache by what the answer depends on, and count what you spend. When you are ready to prove it, take the APV-100 certification exam.

Finished the lesson?Mark it complete to track your progress toward the certification.

Next: Exam APV-100 →

Chain APIs into a summarise-and-score pipeline