Generate RSS feed for any website using CSS selectors

CSS selectors were more useful before the Tailwind fad of dropping meaningful classes names in favor of recreating inline styles but with abbreviations to memorize. I use μBlock Origin + userStyles a lot which both also uses CSS selectors & the last couple of years everything has become a lot harder on the end user to tweak/fix. If you’re lucky now, you’ll have some ARIA attributes to select on.

RSSHub[0] is in the same ballpark, but consists of a large library of site-specific code[1][2].<p>[0]<a href="https:&#x2F;&#x2F;github.com&#x2F;DIYgod&#x2F;RSSHub&#x2F;">https:&#x2F;&#x2F;github.com&#x2F;DIYgod&#x2F;RSSHub&#x2F;</a><p>[1]<a href="https:&#x2F;&#x2F;github.com&#x2F;DIYgod&#x2F;RSSHub&#x2F;tree&#x2F;master&#x2F;lib&#x2F;routes">https:&#x2F;&#x2F;github.com&#x2F;DIYgod&#x2F;RSSHub&#x2F;tree&#x2F;master&#x2F;lib&#x2F;routes</a><p>[2]<a href="https:&#x2F;&#x2F;github.com&#x2F;DIYgod&#x2F;RSSHub&#x2F;tree&#x2F;master&#x2F;lib&#x2F;v2">https:&#x2F;&#x2F;github.com&#x2F;DIYgod&#x2F;RSSHub&#x2F;tree&#x2F;master&#x2F;lib&#x2F;v2</a>

It ded.<p>Archive: <a href="https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20230714202418&#x2F;https:&#x2F;&#x2F;rss-bridge.org&#x2F;bridge01&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20230714202418&#x2F;https:&#x2F;&#x2F;rss-bridg...</a><p>Sample feed: <a href="https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20230308160413&#x2F;https:&#x2F;&#x2F;rss-bridge.org&#x2F;bridge01&#x2F;?action=display&amp;bridge=ABCNewsBridge&amp;topic=act&amp;format=Html" rel="nofollow noreferrer">https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20230308160413&#x2F;https:&#x2F;&#x2F;rss-bridg...</a>

I was always afraid to use on of these. I thought that the css selectors would be too brittle and ultimately break.<p>I have build my own solution that is automagical at <a href="https:&#x2F;&#x2F;awesomegoat.com&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;awesomegoat.com&#x2F;</a> but I am running into next set of issues which are various scraping protections. It seems that reasonable RSS gateway today needs to include botnet of residential proxies just to read content on the internet.

This is a great tool! Before I learned about nitter, this was my primary way to follow people on Twitter. I love the idea of trying to wrestle unsupported feeds (Twitter, Instagram, etc.) into a standard&#x2F;open format.

The lack of feed generation is why I so many of the latest blog platforms are non-starters in my book. It boggles my mind. Honestly, if you don&#x27;t generate a feed of some sort, I really can&#x27;t take you seriously.

I run my own instance of RSS Bridge to keep track of authors that I like on Goodreads.<p>It works pretty well, although every once in a while Goodreads hiccups, and then RSS bridge gives me a bunch of &quot;new posts&quot; that are actually error messages.

Huginn is an another useful tool that allows you to wrangle CSS selectors and XPath nodes to create RSS feeds.<p>I use it quite successfully to get data out of undocumented APIs and out into RSS.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;huginn&#x2F;huginn">https:&#x2F;&#x2F;github.com&#x2F;huginn&#x2F;huginn</a>

This honestly is standard web scraping but these projects always catch my attention.<p>You&#x27;re bound at the mercy of rate-limiting firewalls (so you&#x27;ll have to rotate proxies if you intend on using this heavily) on top of the standard CloudFront bot detection recaptcha, and div-obfuscation (a good example of this is Facebook).

These days I just let chagpt generate a script that scrapes a site and spits out an rss file. Then I run it with cron.

Other services like this: <a href="https:&#x2F;&#x2F;www.fivefilters.org&#x2F;feed-creator&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.fivefilters.org&#x2F;feed-creator&#x2F;</a>

What&#x27;s the easiset way to also run a few basic filters on the site&#x2F;RSS feed&#x27;s content to make it truly shine vs simplistic scraping, like<p>- splitting the full feed by theme of the article into separate feeds and at the same time<p>- remove a few keywords and also<p>- get article length and split into a long &#x2F; short feed<p>- Or maybe get what you used to have on some news sites - subscribe only to a specific author instead of getting bombarded with hundreds of items in a feed

I&#x27;ve wondered why people have tried all sorts of cumbersome ways to splice metadata onto HTML like RDFa but never tried the obvious approach of basing extraction rules on CSS selectors... Often these work without the cooperation of the target site so long as they use CSS the way it was supposed be used (e.g. not tailwind, bootstrap, etc.)

For me PolitePol is best because if doesn&#x27;t limit the amount of feeds and the free plan is pretty good: <a href="https:&#x2F;&#x2F;politepol.com" rel="nofollow noreferrer">https:&#x2F;&#x2F;politepol.com</a>

I wonder if this would work better &#x2F; be more expressive with XPATH-style selectors?

Is there a standalone application that can do similar. That doesn&#x27;t require a web server to run. Like an RSS reader you&#x27;d run on you desktop or phone? I&#x27;d definitely be interested in that.

FreshRSS has XPath scraping.<p><a href="https:&#x2F;&#x2F;danq.me&#x2F;2022&#x2F;09&#x2F;27&#x2F;freshrss-xpath&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;danq.me&#x2F;2022&#x2F;09&#x2F;27&#x2F;freshrss-xpath&#x2F;</a>

Does it work for websites that fetch content async? I&#x27;ve had success with <a href="https:&#x2F;&#x2F;morss.it" rel="nofollow noreferrer">https:&#x2F;&#x2F;morss.it</a> instead (which can also be selfhosted)

This is very similar to how you can scrape data from web with powerquery

Getting 502 Bad Gateway

FreshRSS has this feature built in. But you can use rss-bridge for far more complicated scenarios too

&quot;Generate RSS feed for any website using CSS selectors&quot;<p>For me, &quot;CSS selectors&quot; always seems like a deceptive term, if it means selecting HTML tag elements. What if the website does not use styling.<p>I read 1000s of websites, including all HN submissions, without using CSS. When I want to extract information from a website, I focus on patterns in the page. They might be HTML, they might be style elements, but they could be anything. I never assume that all websites will wrap the information I want in certain elements. There is a ridiculous amount of random variation amongst websites.