RSS to JSON API and XML Sitemap Generator in Node.js and Python

Everything in the previous three lessons reads a page that was built for people and tries to recover the data from it. Feeds and sitemaps are the opposite: files that sites publish specifically so machines can find their content. When one exists, using it is cheaper, more stable and more polite than scraping. A redesign that breaks every scraper you wrote leaves the feed untouched, because the feed is a promise the publisher made to every reader app and aggregator that consumes it.

This lesson covers both directions. Reading other people's feeds to track new content, and generating a sitemap so search engines can find yours.

Feeds are the polite alternative

A typical "monitor these competitors' blogs" job, done by scraping, looks like this: fetch each blog's index page every hour, extract the post links with Link Scraper, diff against yesterday's list, then fetch and convert each new post. That is several calls per site per hour, a parser that breaks when the theme changes, and a crawler hitting someone's server more than it needs to.

The same job with a feed is one call per site per poll. The feed lists the recent posts with titles, dates, summaries and links, in a format that has not changed meaningfully in two decades. You only fetch the full article when a post is new and you actually need its text.

Feeds come in two formats, RSS and Atom. They differ in element names and date formats, and readers handle both. You can usually find a site's feed in one of three ways:

In the page head. Sites advertise feeds with <link rel="alternate" type="application/rss+xml" href="..."> (or application/atom+xml). View the source of the home page or blog index and search for alternate.
At a conventional path. /feed, /feed/, /rss, /rss.xml, /atom.xml, /index.xml and /blog/feed cover a large share of sites. WordPress sites almost always answer at /feed/.
From the platform. Many hosted platforms expose feeds per author, category or tag, which are often more useful than the main feed.

RSS to JSON

RSS to JSON takes a feed URL and returns source (the feed's host), articles (how many items were parsed), maxReached, and feed[], where each item has website, title, pubDate, description and link. It costs 10 credits.

Try it liveRSS to JSON10 credits

POSThttps://api.apiverve.com/v1/rsstojson

curl -X POST "https://api.apiverve.com/v1/rsstojson" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://www.nasa.gov/rss/dyn/breaking_news.rss"}'

url

Each item in feed has title, pubDate, description and link. pubDate is a string in whatever format the feed uses.

It reads Atom as well as RSS: The Verge's index.xml is an Atom feed and came back in the same shape. That is convenient, and it hides a difference you have to handle yourself.

pubDate is a string, not a date

The NASA feed (RSS) returned "pubDate": "Thu, 22 Oct 2026 16:20:37 +0000", the RFC 822 format RSS uses. The Verge feed (Atom) returned "pubDate": "2026-10-22T18:15:00.000Z", ISO 8601. Same field, two formats, depending on the source. Sorting them as strings puts Thursday after Wednesday only by luck. Parse every date into a real timestamp at the boundary.

In JavaScript, new Date(value) parses both of those formats. In Python, datetime.fromisoformat handles neither reliably on 3.9 (it rejects the trailing Z before 3.11), and email.utils.parsedate_to_datetime handles only RFC 822, so try one and fall back to the other. Always keep the timezone; a feed that publishes in -0700 and one that publishes in Z sort correctly only as aware datetimes.

description is usually a teaser

Most feeds publish a summary, not the full article. The GitHub changelog feed ends each description with … followed by "The post ... appeared first on The GitHub Blog." The Verge's descriptions stop mid-word with "… Read the full story at The Verge." Some descriptions contain leftover markup or tab characters. Treat description as a preview for display and triage. When you need the full text, follow link and use URL to Markdown from lesson one, and only for the items that pass your filter.

Check yourself

Your aggregator sorts items from many feeds by `pubDate` as strings and the order is visibly wrong. What is the cause?

Polling without duplicates

A feed is a sliding window: the most recent N items, with old ones dropping off the bottom as new ones appear. Polling it means calling on a schedule and keeping only what you have not seen before. Three rules make that reliable.

Key on link, not on date or title. Titles get corrected after publication. Dates get edited, and some feeds bump pubDate when a post is updated. The link is the closest thing to a stable identity. Normalise it first: drop the fragment and tracking parameters so the same post does not arrive twice under different URLs.

Poll as often as the source changes, not as often as you can. A blog that posts weekly needs a daily poll at most. A news site justifies every 15 to 30 minutes. Polling a weekly blog every five minutes costs you roughly 8,600 calls a month to learn nothing, and the publisher sees an aggressive client.

Notice when you fall behind. If every item in a poll is new, you may have missed items that scrolled out of the window between polls. Log it and shorten the interval for that feed.

// Node 18+, ESM. Stores seen links in a JSON file; use a database in production.
import { readFile, writeFile } from 'node:fs/promises';
import { apiverve } from './apiverve.mjs';

const FEEDS = [
  'https://www.nasa.gov/rss/dyn/breaking_news.rss',
  'https://github.blog/changelog/feed/',
];
const STATE = 'seen.json';

function normalise(link) {
  const u = new URL(link);
  u.hash = '';
  for (const k of [...u.searchParams.keys()]) {
    if (k.startsWith('utm_') || k === 'fbclid' || k === 'gclid') u.searchParams.delete(k);
  }
  return u.href;
}

const seen = new Set(JSON.parse(await readFile(STATE, 'utf8').catch(() => '[]')));
const fresh = [];

for (const url of FEEDS) {
  let data;
  try {
    data = await apiverve('rsstojson', { url });
  } catch (e) {
    console.warn(`skip ${url}: ${e.message}`);
    continue;
  }
  const items = data.feed.map((item) => ({ ...item, link: normalise(item.link) }));
  const unseen = items.filter((item) => !seen.has(item.link));
  if (items.length && unseen.length === items.length && seen.size) {
    console.warn(`${data.source}: every item is new, some may have been missed`);
  }
  for (const item of unseen) {
    seen.add(item.link);
    fresh.push({ ...item, published: new Date(item.pubDate) });
  }
}

fresh.sort((a, b) => b.published - a.published);
for (const item of fresh) console.log(item.published.toISOString(), item.website, '-', item.title);
await writeFile(STATE, JSON.stringify([...seen]));

import json
import os
from datetime import datetime
from email.utils import parsedate_to_datetime
from urllib.parse import urlparse, urlunparse, parse_qsl, urlencode

from apiverve import apiverve

FEEDS = [
    "https://www.nasa.gov/rss/dyn/breaking_news.rss",
    "https://github.blog/changelog/feed/",
]
STATE = "seen.json"

def normalise(link: str) -> str:
    p = urlparse(link)
    q = [(k, v) for k, v in parse_qsl(p.query) if not k.startswith("utm_") and k not in ("fbclid", "gclid")]
    return urlunparse(p._replace(query=urlencode(q), fragment=""))

def parse_date(value: str) -> datetime:
    try:
        return parsedate_to_datetime(value)                        # RSS: RFC 822
    except (TypeError, ValueError):
        return datetime.fromisoformat(value.replace("Z", "+00:00"))  # Atom: ISO 8601

seen = set(json.load(open(STATE))) if os.path.exists(STATE) else set()
fresh = []

for url in FEEDS:
    try:
        data = apiverve("rsstojson", {"url": url})
    except RuntimeError as e:
        print(f"skip {url}: {e}")
        continue
    for item in data["feed"]:
        link = normalise(item["link"])
        if link in seen:
            continue
        seen.add(link)
        fresh.append({**item, "link": link, "published": parse_date(item["pubDate"])})

for item in sorted(fresh, key=lambda i: i["published"], reverse=True):
    print(item["published"].isoformat(), item["website"], "-", item["title"])

with open(STATE, "w") as f:
    json.dump(sorted(seen), f)

Before Python 3.11, fromisoformat does not accept the trailing Z that Atom feeds use, which is why the code swaps it for +00:00 first. The three-digit milliseconds in The Verge's dates are fine on 3.9; other fractional-second lengths are not, so if you meet a feed that writes them, trim them before parsing.

When a URL is not a feed

Pointing RSS to JSON at an ordinary web page fails rather than returning an empty list. Sending https://example.com produced an error explaining that the feed "could not be read", with the XML parser's complaint about the HTML. That makes it usable as a discovery probe: try the conventional paths in order and keep the first one that succeeds. Cap the number of guesses, since each attempt is a request, and cache the answer per site for a long time, because sites rarely move their feeds.

Check yourself

Which identity should an RSS poller use to decide whether an item has already been processed?

Answer the check above to continue.

Sitemaps: the other machine-readable map

A sitemap is an XML file listing a site's URLs, optionally with a last-modified date, a change frequency and a priority for each. Search engines read it to discover pages they might not reach by following links. For you as a data engineer, sitemaps matter in two directions.

Reading someone else's. A site's sitemap is a list of its pages that the site wants found. It usually lives at /sitemap.xml, and robots.txt often names it with a Sitemap: line. Large sites split it into a sitemap index that points to several child sitemaps. When you need the URLs of a site's product pages or articles, the sitemap is a far better starting point than crawling: one or a few files instead of thousands of page fetches. It is XML; fetch it with your own HTTP client and parse it, or turn it into JSON with XML to JSON. The lastmod values, when a site maintains them honestly, tell you which pages changed since your last pass.

Generating your own. That is what Sitemap Generator does. It does not crawl anything. You send a list of URLs and it returns a valid sitemaps.org XML document. It costs 2 credits.

Try it liveSitemap Generator2 credits

POSThttps://api.apiverve.com/v1/sitemapgenerator

curl -X POST "https://api.apiverve.com/v1/sitemapgenerator" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls":["/","/pricing","/docs/getting-started"],"baseUrl":"https://example.com","includeLastMod":false}'

urlsbaseUrlincludeLastMod

Relative paths are joined to baseUrl. urlCount is the number of url entries written.

The response gives you sitemap (the XML string), urlCount, format (XML), protocol (sitemaps.org/0.9), and a settings object echoing defaultChangeFreq, defaultPriority, includeLastMod and baseUrl. sizeBytes and sizeKb are premium.

Per-URL settings are passed by sending objects instead of strings in urls, each with url and any of priority, changeFreq and lastMod; anything you leave out uses the defaults from the top-level changeFreq (default weekly) and priority (default 0.5).

Two defaults to override

includeLastMod is true by default, and the date it writes is today. Generating a sitemap for two URLs with the default produced <lastmod>2026-10-22</lastmod> on both, which was the day of the call, not the day either page changed. Regenerate nightly and every page on your site claims to have changed every night. Search engines learn to ignore lastmod from sites that do that, which throws away the one signal that could get your updated pages recrawled sooner. Either pass a real lastMod per URL from your CMS or database, or set includeLastMod to false.

It does not deduplicate. Sending /pricing twice produced two identical <url> entries and a urlCount of 4 for four inputs. Duplicates in a sitemap are sloppy at best. Deduplicate, and normalise trailing slashes consistently, before you call.

build-sitemap.mjs

// Node 18+, ESM
import { writeFile } from 'node:fs/promises';
import { apiverve } from './apiverve.mjs';

// In a real site these come from your database or CMS, with real modification dates.
const pages = [
  { path: '/', updated: '2026-10-01' },
  { path: '/pricing', updated: '2026-09-14' },
  { path: '/pricing', updated: '2026-09-14' },        // duplicate, removed below
  { path: '/docs/getting-started', updated: '2026-10-20' },
];

const unique = [...new Map(pages.map((p) => [p.path, p])).values()];

const data = await apiverve('sitemapgenerator', {
  baseUrl: 'https://example.com',
  urls: unique.map((p) => ({ url: p.path, lastMod: p.updated })),
  changeFreq: 'monthly',
});

await writeFile('sitemap.xml', data.sitemap);
console.log(`wrote ${data.urlCount} URLs`);

The sitemaps.org protocol caps a single sitemap at 50,000 URLs and 50 MB uncompressed. Past that, split into several files and publish a sitemap index that lists them. Then reference the sitemap (or the index) from robots.txt with a Sitemap: line, and submit it in Google Search Console and Bing Webmaster Tools so you can see how many URLs were accepted.

Choosing your source

You now have every way to get content from the web covered in this course, and the order to try them in is roughly the order of politeness:

A feed, when you want to know what is new. One small call per poll, stable format.
A sitemap, when you want the full list of a site's pages. One file, maintained by the site.
Metadata, when you want to know what a page is. Two credits, no body text.
Text or Markdown conversion, when you need the content itself, for the specific pages you actually need.
Screenshots, when only the rendered result answers the question.

Every step down that list costs more and asks more of the site you are reading. Starting at the top and going only as far as the task needs is both the cheapest design and the one that keeps you welcome.

That completes Extract Web Content. The next course in the Web Data Developer certification moves from reading pages to judging them, with SEO checks, tech stack detection, DNS, SSL and domain intelligence: Analyze Websites.

Finished the lesson?Mark it complete to track your progress toward the certification.

Next: Build an automated SEO audit with two APIs →

RSS feeds and sitemaps

Feeds are the polite alternative

RSS to JSON

pubDate is a string, not a date

description is usually a teaser

Polling without duplicates

When a URL is not a feed

Sitemaps: the other machine-readable map

Two defaults to override

Choosing your source

Run every example on your own key. Free plan, no card.

Scaling up?