Reddit's robots.txt disallows all web crawlers

8 pointsby rawfael11 months ago

1 comment

rawfael11 months ago

Seems that they might be showing a different robots.txt to Google: <a href="https://x.com/pedrodias/status/1808799403798409216" rel="nofollow">https://x.com/pedrodias/status/1808799403798409216</a>