RSS.expert

· PM-mirror

I had owned rss.expert for three years without putting it to any use. Every year the renewal came around, I paid it, and the domain stayed right where it was: no website, just waiting for an idea that never arrived. A name like that practically demands something involving feeds. The tricky part was figuring out what.

For anyone who's never had reason to know: RSS is a file a website publishes listing its latest updates. Titles, links, dates, sometimes the entire text. A feed reader pulls those files together from different websites so you can follow everything in one place, without an algorithm deciding the order. Atom and JSON Feed are RSS's cousins; they do the same job with different syntax. The technology dates back to the 2000s. Plenty of people declared it dead when Google Reader shut down in 2013, yet plenty of blogs still publish feeds.

In September 2026, I finally sat down and did something with the domain. The result is live at rss.expert.

A website your terminal can read, too #

The idea I settled on fits in one sentence: the same address responds to both your browser and your terminal. Open rss.expert/weird in a browser and you see a webpage. Run this in a terminal:

curl https://rss.expert/weird

and you get the same content as neatly aligned plain text, without an HTML tag in sight:

THE WEIRD WEB

─────────────────────

Small feeds, personal sites, and the odd corners of this index.

01. Bear Roll

    bearroll.dev

    https://bearroll.dev/

[...]

04. linkhut

    ln.ht

    https://ln.ht/

This works because of an HTTP detail almost nobody pays attention to. Every request carries a header called Accept, which tells the server what kind of response the client prefers. A browser sends text/html. curl sends */*, meaning “anything will do.” The server reads that header and chooses: HTML for one, text for the other. That's why the project doesn't have a separate API. The address you open in your browser is the very same one you call from a script.

You can force the format in three ways: with a file extension (/latest.json, /latest.rss, /latest.atom, /weird.txt), the ?format= parameter, or the Accept header itself. If none of those specifies a format, the response is plain text. Ask for a format the page doesn't support and you get a 406 error, which is readable in the terminal, too.

For people who practically live in the terminal, the most useful option is format=urls. It returns nothing but addresses, one per line. It exists so you can drop rss.expert into a pipe:

curl -s "https://rss.expert/search?q=openbsd&format=urls"

There's a catch on Windows. In PowerShell, curl can be an alias for Invoke-WebRequest, which returns an object instead of plain text. Use curl.exe.

The commands #

Each command is a short URL, and its explanation lives at /help/name. /weird returns the list; /help/weird explains how that list is put together. The homepage has two lines and an example, because everything else is in /help.

Some commands are for wandering around what the service has already collected:

The others work on any URL you give them:

These endpoints also accept POST requests containing HTML you've already downloaded. In that case, the service doesn't need to fetch the page again:

curl https://example.org | curl -X POST --data-binary @- https://rss.expert/extract

No model, just weights and arithmetic #

The selections for /weird and /one are heuristic. There's no AI model making the choices, and /about says so explicitly. It's a points system, with all the weights defined at the top of one file: Ranking.php.

Rare domains, small feeds, personal-site vibes, projects hosted on Codeberg, mentions of Gemini or Gopher (two alternative protocols beloved by the small web), custom domains, and recently seen feeds all earn points. Big portals, aggregators, SEO-looking titles, and domains that already appear too often in the index lose points. The score doesn't dictate the final list: it only makes a link more likely to be drawn, so each request returns a different handful.

/trending compares titles from the last 48 hours with those from the previous 14 days, and only shows a term if it appeared at least three times in the recent window. The page also makes it absolutely clear that these trends describe the rss.expert index, not the whole internet. I'd rather spell that out than have someone mistake it for a list of what the entire world is talking about today.

The starting point is the catalog at weirdnet.org, another project of mine: an observatory of the small web with curated links. The entire catalog is included in the repository and imported into the database during installation. When I wrote this, it held 37,118 entries.

From there, there's automatic discovery, designed to be deliberately timid. When a new article comes in, the service records up to five external links from it. Later, another process visits a small batch of those pages, looks for a feed, validates it, and only then stores it. It doesn't follow links from the pages it visits, so discovery takes one hop and stops. A domain visited in the last 24 hours won't be revisited, and obvious social networks are excluded. The idea is to stumble upon a neighbor now and then, not crawl the web.

The same restraint applies when fetching feeds. Every request sends ETag and If-Modified-Since, the HTTP way of asking a website, “Has anything changed since last time?” If the response is 304 (not modified), the service counts it as a success and spaces out the next visit, up to 12 hours. If a feed keeps failing, the wait doubles after every failure, capped at seven days. The bot identifies itself as RSSExpert/1.0 with a link to /about, so anyone spotting it in their logs can tell who it is and what it wants.

Cron runs the fetch job every ten minutes, processing no more than 20 feeds at a time. Five minutes would be too short: 20 feeds with a ten-second timeout each could exceed that on a bad run. A lock keeps two runs from stepping on each other.

How I built it #

The Git history tells the story without much shame: three consecutive commits called “first commit,” all dated September 26, followed the next day by one called “redesign 3.” Redesigns 1 and 2 are nowhere to be found.

The stack is PHP 8.3, SQLite, Nginx with PHP-FPM, and cron. No Laravel, Symfony, Node, Redis, Elasticsearch, or Docker. PHP's standard facilities already handle HTTP (through the curl extension), XML, HTML, JSON, and SQLite, and Composer is there only to autoload classes. The router is a few dozen lines long and lives in the repository itself. For a service this size, a framework would give me more code to maintain than code to write.

SQLite stores everything: feeds, articles, fetch history, tags, discovered URLs, and even rate-limit state, which would normally go in Redis. Search uses FTS5, the full-text index built into SQLite, covering titles, content, summaries, and authors. The service's entire state lives in one file, and backing it up means copying that file using SQLite's own .backup command.

Feeds are messy, and deduplication has to account for that. An article is first identified by its GUID, if the feed supplies a usable one (the code discards values like null and undefined, which crop up more often than they should). Without a GUID, it uses the normalized URL, stripped of its fragment and default port. If neither is available, it falls back to a hash of the URL, title, and text.

The HTML side takes a similar approach. Each page's text and HTML are generated from the same data structure, and the template is just a shell: a skip-navigation link, a short menu, and the footer. That way the terminal and browser never show different information. The homepage loads no fonts, scripts, or images.

The boring part that had to be right #

A service that fetches any URL a stranger submits has a well-known risk called SSRF. Instead of requesting a public website, someone submits http://localhost or the IP address of a machine on the internal network. The server, sitting on the inside, fetches it and returns the response.

Every incoming URL goes through a class called UrlGuard before a connection is made. It rejects any scheme other than http or https, URLs containing usernames and passwords, localhost, .local, .internal, and cloud metadata service hostnames. It also rejects private, loopback, CGNAT, and multicast IP addresses, including shortened representations that fool naive validators, such as 127.1 or an address written as a decimal integer. If DNS returns several addresses and even one is internal, the entire URL is rejected, regardless of whether the others are public.

There are two further precautions. The validated IP is pinned to the connection, so DNS can't change its answer between validation and the request, and the service then checks which IP it actually connected to. Every redirect is checked again, with a maximum of three. The tests cover a public website redirecting to a private address.

XML is parsed without network access, HTML from feeds is escaped before being displayed, and endpoints that fetch remote URLs have per-IP request-per-minute limits. The log stores the route, status, duration, and error. It doesn't record the URL the visitor requested.

The tests don't use the network. They run entirely against fixture files stored in the repository, covering RSS, Atom, and JSON Feed parsers, URL normalization, feed discovery, deduplication, search, content negotiation, and SSRF safeguards.

Where things stand today #

When I finished writing this, /status said: one feed, 37,118 articles, last fetch: never. The only feed is the Weirdnet catalog, which gets imported rather than fetched by cron. In other words, the bot I spent so much time teaching to be polite hasn't knocked on anyone's door yet.

But at least now, when the domain renewal comes around, I know what I'm paying for.