TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Amazon's AI crawler is making my Git server unstable

3 点作者 subset4 个月前

3 条评论

LinuxBender4 个月前
Nginx has an assortment of options to rate limit people&#x2F;bots. [1] For the 10% not using a user-agent generic rate limits could be applied based on volume <i>limit_rate_after</i>. One would have to get creative with nginx maps. Another option would be to calculate patterns in your access logs and just blackhole or ipset reject IP addresses or networks that are abusive accepting some of them may be abusive humans vs abusive bots.<p>[1] - <a href="https:&#x2F;&#x2F;serverfault.com&#x2F;questions&#x2F;639671&#x2F;nginx-how-to-limit-request-rate-based-on-user-agent" rel="nofollow">https:&#x2F;&#x2F;serverfault.com&#x2F;questions&#x2F;639671&#x2F;nginx-how-to-limit-...</a>
osdotsystem4 个月前
I feel for you. Same for OpenAI (<a href="https:&#x2F;&#x2F;www.linkedin.com&#x2F;posts&#x2F;appinv_openai-is-a-felon-company-with-pirate-altman-activity-7286300344869949440-WI8I" rel="nofollow">https:&#x2F;&#x2F;www.linkedin.com&#x2F;posts&#x2F;appinv_openai-is-a-felon-comp...</a>). A friend told me Ai bots even ignored the robots.tx ...
评论 #42749362 未加载
fragmede4 个月前
&gt; What else do I need to do?<p>I mean, not sending 418 and sending 429 or 403 or anything more useful would be on my list of things to try. Might double check what 418 is while you&#x27;re trying to figure out why sending that doesn&#x27;t seem to do anything.
评论 #42749037 未加载