I'm not sure how I feel about the content of this entire post. RSS feeds are a more or less stagnant technology, mostly adopted "for fun" or by a niche of people who find them useful. The only way I can see to move forward is to make them as easy and painless to use as possible, the onus falls on both the creator and consumer of feeds (the websites and the clients... the user is completely out of the equation here in my opinion).<p>This kind of attitude reeks of the "you're using it wrong!" of the Linux world. Are you really telling me you are getting enough RSS feed requests to put a dent on your tech stack? Is your bandwidth overhead suffering that much (are you not caching?)? Make it painless and let's be thankful it isn't all web scrapers masquerading as users.<p>Mind-boggling problem to be angry about.
I love RSS but this approach worries too much.<p>It's static content, at least as static as the rest of your content, so write once and let one of a dozen CDNs cache it for you for free. They will do a good job of setting cache headers for you. If people want to ignore them and honk on it from their crappy readers, that's on them.<p>Even if <i>you're</i> personally serving it from your kitchen toaster, it's static content. You need a significant number of bad actors for this to be a problem.<p>Moreover, if this post is anything to go by, treating a bad reader with a logical hurdle and expecting a sane result is baffling.<p>It's not worth this much headspace.
> There's another one which hits every 2 minutes without fail [...] it's the one feed user-agent I've had to block outright. It doesn't stop requesting things [...] Clearly, I need to use a bigger hammer.<p>I'd be tempted to go the opposite direction and give it articles that seem new, every time, called "Your feed reader is very broken and you should use a different one".
I checked what my feed reader does. It seems that it typically gets 304s (I send both If-None-Match and If-Modified-Since) but still about half of the requests get 429s.<p>So the request pattern basically looks like:<p>1. Get 304.<p>2. Wait 5min.<p>3. Get 429.<p>4. Error backoff evenly distributed between 30-60min<p>5. Go back to step 1.<p>So I guess 5min is "too fast" even for conditional requests. However the reason my reader picks 5min is:<p>1. There is no caching headers to suggest the author's preference.<p>2. Conditional requests are supported.<p>3. The feed is fast.<p>4. The feed is popular.<p>5. The feed is active (posts every week or so)<p>Sure, for her feed it could probably be lower but how is a robot supposed to know that? I would highly recommend setting a cache header. That gives an automatic signal of when the last fetch should be considered stale. I'll admit that many feed readers just use a fixed schedule, but many will still use a library that accidentally caches requests (maybe the browser extension readers that she was complaining about would?). For my feed reader we won't poll more often than your cache header (Cache-Control or Expires will both work) unless it is more than 24h in which case we will poll no more frequently than daily. If you just post blog posts about it it is never going to change, it is better to start pushing for broader support of an actual protocol that could be implemented.<p>Another great option is supporting WebSub. It is easy to get set up with a public hub and then readers that support it will poll very rarely. (Mine will poll weekly.)
Granted I do not have any blog feed, much less a reasonably popular one like Rachel by the Bay, but I would have thought that serving simple static content like an RSS feed is quite low-cost such that it almost doesn't matter how much people are querying, no? I'd be quite surprised if the author's server was genuinely struggling under the weight of these requests but happy to be corrected.
I was surprised to hear that they got authors of RSS readers to get in touch and fix issues. Last time I looked for an RSS reader for Windows, it was a graveyard of abandoned software.
You cannot hope that consumers are well-behaved on the Internet, it has never been the case unfortunately, you have to act defensively.<p>If the issue is the processing power because the content served is dynamically generated, cache the content. Even with a server running on ESP32, just a few second should be enough (if the rest of the website to be statically generated)<p>If the issue is the bandwidth you may focus on big content (i.e. media) and return a 429 or use a CDN just for that (not worth for a feed in my opinion)<p>If the issue is neither, why bother ? Your time is precious, and you should spend it on more enjoyable things (that is, unless you find doing this enjoyable of course).
On the topic of RSS feeds. I really do miss this way of consuming news.<p>I tried using a few RSS feeds from major news organizations (WaPo, NYT). All I get are the headlines and a link to the article. I am not sure if it has always been this way or not.<p>On the other hand, “rachelbythebay” blog posts can be viewed entirely in the RSS reader.<p>I am using “feedly” on iOS. Downside I see is that it does not properly display all of the formatting elements. Some examples:<p>- Bullets represented as just *.<p>- The “code” sections sort of sit awkwardly in the reader. No formatting applied.<p>- ~~In some of the code sections, it’s just awkwardly replaced by ellipses (…)~~ nvm, this is the authors writing style :)<p>Maybe that’s just a specific RSS reader issue?
>I did that way back then because browsers used to care about RSS and Atom, and they'd put that little yellow feed icon somewhere in the top bar when they spotted this sort of thing in a page. At least in the case of Firefox, you could click on it, and it would throw the target URL to a helper of your choice.<p>Still here in Vivaldi.
> A fair number of people are sending conditional requests, but are doing it every 5 or 10 minutes. This is ridiculous. I don't write that often, and never have.<p>This objection might make more sense if the purpose of a feed reader was to do nothing except check rachelbythebay.com for updates.
The Internet is dying. Everything shifts from standards toward managed corporate walled gardens. There is no place for RSS in the future. This is my personal opinion.<p>I use only RSS. That is how I obtain new information. I do not know personally anyone else doing that. I ping sources every hour, but I ping at least 400 sources.<p>From my sources none provided last-modified in headers: reddit, youtube, personal sites (In ff f12, network). My site supports it. It is nice also that you support it, but I doubt there is any impact of that in real world. Most of the attention goes through tiktok,youtube videos, through chrome browser. Either it is supported now, or it doesn't really matter if 40 dudes makes request every minute or so.<p>We should also provide clean title, description, in open graph protocol meta data, and yet not everybody does that.<p>We should also return correct HTTP status codes, and yet not everybody does that.<p>I am disenchanted with current state of the Internet, or maybe it was always a little bit pile of various things/garbage.
So much effort for such specific pedantry.<p>I just added her feed to a third party proxy service. If the owner of the site doesn’t want to serve a very small and basically static text file in a way people expect then that is their prerogative.<p>But having to run their feed through a proxy to work around this doesn’t seem to be something they would like to encourage.