Clean Scraped Text: Language, Keywords and Summaries via API

Extracted text is raw material, not data. A page run through Website to Text gives you one long string: the article, yes, but also a cookie notice, a "related stories" rail, and a footer with the copyright year. Before it is useful to anything downstream you need to know what language it is in, what it is about, what it says in brief, and how someone will find it again. Each of those is one API call. The work is in knowing when each call's answer can be trusted.

The target is a record like this, built from one page:

record.json

{
  "url": "https://example.com/post",
  "language": "en",
  "languageSure": true,
  "summary": "Two or three sentences lifted from the text.",
  "keywords": ["crawler", "server", "robots"],
  "searchKey": "polite crawling respects robotstxt caches responses",
  "text": "the cleaned body text, kept for reprocessing"
}

Keep text. Every field above it is derived, and the day you change how you derive them you will want to rebuild from the source rather than re-fetch every page.

When you have HTML, not a URL

Most of this course extracts from a URL. Sometimes the HTML arrives another way: an RSS item's description field, a CMS export, a webhook from a form builder, a page you rendered yourself. Sending that to HTML to Text costs 2 credits and skips the fetch entirely.

Try it liveHTML to Text2 credits

POSThttps://api.apiverve.com/v1/htmltotext

curl -X POST "https://api.apiverve.com/v1/htmltotext" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"html":"<nav>Home | About</nav><article><h1>Release notes</h1><p>Version 2.4 adds retries.</p><p>Version 2.3 fixed a crash.</p></article><footer>Copyright 2026</footer>"}'

html

Look at the joins between elements in text. wordCount and characterCount come back on every plan; detectedLanguage is premium.

Run that and read text closely. When we called it, the result was Home | AboutRelease notesVersion 2.4 adds retries.Version 2.3 fixed a crash.Copyright 2026. Two things to notice. Block elements are joined with no whitespace, so About and Release fuse into one token, and so do crash. and Copyright. And the navigation and footer came through, because extractionMethod was markup: on a fragment this small there is no article body to isolate.

Both are fixable before the call, cheaply and locally. Remove the chrome you know you don't want, and put a newline after every block-level closing tag so words cannot fuse:

prep-html.js

const BLOCK = /<\/(p|div|h[1-6]|li|tr|td|th|section|article|header|footer|blockquote|pre|br)\s*>/gi;

export function prepHtml(html) {
  return html
    .replace(/<(nav|footer|aside|script|style|form)[\s\S]*?<\/\1>/gi, '')
    .replace(/<br\s*\/?>/gi, '\n')
    .replace(BLOCK, (m) => m + '\n');
}

A fused token is the kind of bug nobody notices until search stops working: a user searches for "release" and the one page that mentions it is indexed under aboutrelease. Fix it at the boundary, before anything else sees the text.

Detect the language, and know when not to believe it

Language decides everything after it. Keyword extraction drops stopwords, and stopwords are language-specific. A summariser that ranks sentences tuned for English will produce something odd from Finnish. A search index needs to know which stemmer to apply. So the first enrichment call is Language Detector, at 2 credits.

Try it liveLanguage Detector2 credits

POSThttps://api.apiverve.com/v1/languagedetector

curl -X POST "https://api.apiverve.com/v1/languagedetector" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text":"Le service est temporairement indisponible. Merci de réessayer plus tard."}'

text

Compare the confidence of the first and second entries in detectedLanguages, not the absolute number.

The answer is correct: primaryCode is fr. Now look at the confidences we got: French 0.300, Latin 0.246, Portuguese 0.235. A correct answer at 0.30. If you had written if (confidence > 0.8), you would throw away nearly every real detection. These scores are relative weights spread across candidate languages, not probabilities of being right. A German sentence came back at 0.376 against Dutch at 0.299, also correct.

Now send the single word Gift. We got vietnamese at 0.227, with Cebuano at 0.219 right behind it. The word is English (and German, where it means poison). There was no signal to find, and the detector still returned a primary language, because that is what the field is for.

So the rule that works is about the margin, plus a minimum length:

Below about 20 characters, don't ask; mark the language as undetermined.
Accept the top answer when it leads the runner-up by a clear margin. In our calls, real text led by 0.05 or more; the one-word guess led by under 0.01.
When the margin is thin, fall back to a declared language if you have one. Website to Text returns the page's own language field, which is what the publisher said the page is in.

The confidenceLevel label (low, medium, high) is premium, so do not build on it if you are on a free key. Also handle an empty code: some candidates in detectedLanguages have no ISO 639-1 code (Cebuano came back with ""), so a primaryCode of "" should be treated as undetermined too.

Detect on a slice, not the whole page. The first 1,000 or so characters of body text are plenty, and sending less is faster.

Check yourself

Language Detector returns primaryCode "en" with detectedLanguages confidences of 0.24, 0.23 and 0.20 for a 15-character product name. What should your pipeline record?

Keywords and summaries: what they are, mechanically

Both enrichment steps here are statistical, not semantic. That is a feature: they are fast, cheap and repeatable, and they never invent a word that was not in the text. It also tells you exactly how they fail.

Keyword Extractor (10 credits) counts terms after removing stopwords and punctuation, and returns keywords (every term with its count), topKeywords (the leading five with a percentage share), topKeyword, keywordCount and totalOccurrences.

Try it liveKeyword Extractor10 credits

POSThttps://api.apiverve.com/v1/keywordextractor

curl -X POST "https://api.apiverve.com/v1/keywordextractor" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text":"Rate limiting protects the server. A polite crawler respects robots.txt, caches responses, and keeps concurrency low. The crawler identifies itself with a contact address so the server owner can reach you."}'

text

Find robots.txt in the keywords object. What happened to the dot?

We got server and crawler at two occurrences each, and robots.txt came back as robotstxt. That is the punctuation removal at work, and it is worth knowing before you show keywords to users or match them against a list. Singular and plural are counted separately, so "pipeline" and "pipelines" are two keywords. On a short text, the top five are mostly ties at a count of one, which tells you nothing. Keywords earn their keep on article-length text, where the counts separate.

Because the stopword list is language-specific, only run extraction on text whose language you trust and that the extractor handles. Gate it on the language step.

Text Summarizer (20 credits) is extractive: it scores sentences and returns the best ones verbatim in summary, with sentences controlling how many (default 3). Word counts and the reduction percentage are premium.

Try it liveText Summarizer20 credits

POSThttps://api.apiverve.com/v1/textsummarizer

curl -X POST "https://api.apiverve.com/v1/textsummarizer" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text":"The robots exclusion standard tells crawlers which paths a site owner does not want fetched. It is a plain text file served at the root of a host. Compliance is voluntary, but ignoring it is the fastest way to get blocked. Many sites also publish a crawl-delay directive. Search engines read the file before every crawl session. A polite crawler caches the file and refreshes it daily.","sentences":2}'

textsentences

Every sentence in summary appears word for word in the input.

Two consequences. Garbage in the input becomes garbage in the summary: if the cookie banner is the most "central" sentence on a thin page, it will be chosen. Clean first. And the summariser takes up to 5,000 characters, so a long article must be cut. Cut at a sentence boundary, and cut from the top: on news and documentation pages the important material is usually early.

Check yourself

Summaries for one site keep starting with "We use cookies to improve your experience." What is the right fix?

Answer the check above to continue.

A search key, and why you keep the original

Search wants text normalised: lowercase, no punctuation, consistent whitespace. Display wants the original. Store both.

Remove Punctuation costs 1 credit and returns cleaned. The list of removed characters and their count are premium.

Try it liveRemove Punctuation1 credit

POSThttps://api.apiverve.com/v1/removepunctuation

curl -X POST "https://api.apiverve.com/v1/removepunctuation" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text":"Don't stop -- it's 3:45, e-mail us!"}'

text

Look at what happened to the apostrophes, the time and the hyphenated word.

We got Dont stop its 345 email us. That is exactly right for a search key: a user typing "dont" or "email" now matches. It is also lossy in ways you must decide on. 3:45 became 345. C++ would become C, and node.js would become nodejs. If your corpus is technical, protect a short list of terms (replace them with placeholders before the call and restore them after), or index both the raw and the cleaned form.

Why call an API for something a regex can do? Consistency. The same 1-credit call gives identical output from your Node indexer, your Python backfill script and a no-code tool, and one definition of "clean" is worth more than saving a credit. If you only ever normalise from one codebase, a local regex is a fine choice, and the Python tab below shows the equivalent.

The cleaning step, end to end

This module takes cleaned body text (from Website to Text, or from HTML to Text after prepHtml) and returns the record. The language gate decides whether the English-tuned enrichment runs. Node 18+, ESM, reusing verve.js from the previous lesson.

import { verve } from './verve.js';

const MIN_DETECT_CHARS = 20;
const MIN_MARGIN = 0.03;
const ENRICH_LANGS = new Set(['en']);

export async function detectLanguage(text, declared = null) {
  const sample = text.slice(0, 1000);
  if (sample.length < MIN_DETECT_CHARS) return { language: declared, languageSure: false };
  const { primaryCode, detectedLanguages = [] } = await verve('languagedetector', { text: sample });
  const [first, second] = detectedLanguages;
  const margin = (first?.confidence ?? 0) - (second?.confidence ?? 0);
  if (primaryCode && margin >= MIN_MARGIN) return { language: primaryCode, languageSure: true };
  return { language: declared || primaryCode || null, languageSure: false };
}

function cutAtSentence(text, max = 5000) {
  if (text.length <= max) return text;
  const cut = text.slice(0, max);
  const end = cut.lastIndexOf('. ');
  return end > max * 0.5 ? cut.slice(0, end + 1) : cut;
}

export async function cleanRecord(url, rawText, declaredLanguage = null) {
  const text = rawText.replace(/\s+/g, ' ').trim();
  const record = { url, text, ...(await detectLanguage(text, declaredLanguage)) };

  if (record.languageSure && ENRICH_LANGS.has(record.language)) {
    const [kw, sum] = await Promise.all([
      verve('keywordextractor', { text: text.slice(0, 50000) }),
      verve('textsummarizer', { text: cutAtSentence(text), sentences: 3 }),
    ]);
    record.keywords = (kw.topKeywords ?? []).map((k) => k.keyword);
    record.summary = sum.summary;
  }

  const head = `${record.summary ?? text.slice(0, 500)}`;
  const { cleaned } = await verve('removepunctuation', { text: head });
  record.searchKey = cleaned.toLowerCase().normalize('NFKC').replace(/\s+/g, ' ').trim();
  return record;
}

import os
import re
import unicodedata
import requests

API = "https://api.apiverve.com/v1"
HEADERS = {"x-api-key": os.environ["APIVERVE_API_KEY"]}


def verve(api, body):
    r = requests.post(f"{API}/{api}", json=body, headers=HEADERS, timeout=30)
    data = r.json()
    if not r.ok or data.get("status") != "ok":
        raise RuntimeError(data.get("error") or f"HTTP {r.status_code}")
    return data["data"]


def detect_language(text, declared=None):
    sample = text[:1000]
    if len(sample) < 20:
        return {"language": declared, "languageSure": False}
    d = verve("languagedetector", {"text": sample})
    cands = d.get("detectedLanguages") or []
    first = cands[0]["confidence"] if cands else 0
    second = cands[1]["confidence"] if len(cands) > 1 else 0
    code = d.get("primaryCode")
    if code and first - second >= 0.03:
        return {"language": code, "languageSure": True}
    return {"language": declared or code or None, "languageSure": False}


def local_search_key(text):
    # The local equivalent of removepunctuation, for single-codebase pipelines.
    text = unicodedata.normalize("NFKC", text).lower()
    text = re.sub(r"[^\w\s]", "", text)
    return re.sub(r"\s+", " ", text).strip()


if __name__ == "__main__":
    print(detect_language("Guten Tag, wie geht es Ihnen?"))
    print(local_search_key("Don't stop -- it's 3:45, e-mail us!"))

Count the cost per page: 2 for language, plus 30 for keywords and summary when the page is English, plus 1 for the search key. Non-English pages cost 3. Adding a language to ENRICH_LANGS should be a deliberate decision after you have looked at the output for that language, not a default.

Clean records are what you store. The next lesson uses the same normalisation idea for a different purpose: deciding whether a page has really changed. Continue with Monitor pages for changes.

Finished the lesson?Mark it complete to track your progress toward the certification.

Next: Monitor web pages for changes →

Clean and structure extracted text

When you have HTML, not a URL

Detect the language, and know when not to believe it

Keywords and summaries: what they are, mechanically

A search key, and why you keep the original

The cleaning step, end to end

Run every example on your own key. Free plan, no card.

Scaling up?