TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Scrapy needs to have sane defaults that do no harm

12 pointsby sagacity27 days ago

1 comment

sph27 days ago
Is there a pluggable fail2ban-like server for HTTP, to rate-limit and block with a 429 any rude bot?<p>Also, how come we haven&#x27;t yet designed a system of pluggable HTTP pipelines? If you have a custom web app, either you implement rate limiting by hand, or you set that up in your frontend nginx&#x2F;caddy&#x2F;apache&#x2F;etc. It would be much nicer if we had standardized composable HTTP server processes that either handle a request, or pass through to the next one. That way rate limiting would just be a matter of plugging a rate-limiter process just before your web app.<p>I guess I just miss having Elixir&#x27;s Plug system at the OS level.
评论 #43772849 未加载
评论 #43780868 未加载