Facebook's robots.txt

<a href="http://disqus.com/humans.txt" rel="nofollow">http:&#x2F;&#x2F;disqus.com&#x2F;humans.txt</a>

<a href="http://www.google.com/robots.txt" rel="nofollow">http:&#x2F;&#x2F;www.google.com&#x2F;robots.txt</a>

You don&#x27;t scrape Facebook, Facebook scrapes you!

So what does it mean by facebook whitelisting a scraping service? Do they actively block scrapers?

Is there a way to replace this robots.txt with a null robots.txt? :)

What is a User Agent: Yeti?

Even Facebook&#x27;s robots.txt has a hatred for my pseudo-anonymous browser settings. Facebook gives me this (for any page): &quot;Sorry, something went wrong. We&#x27;re working on getting this fixed as soon as we can.&quot;