科技回声

7 条评论

perryh2超过 11 年前

<a href="http://disqus.com/humans.txt" rel="nofollow">http://disqus.com/humans.txt</a>

评论 #6822970 未加载

评论 #6821761 未加载

评论 #6821904 未加载

viana007超过 11 年前

<a href="http://www.google.com/robots.txt" rel="nofollow">http://www.google.com/robots.txt</a>

评论 #6823056 未加载

评论 #6821679 未加载

kr1m超过 11 年前

You don't scrape Facebook, Facebook scrapes you!

评论 #6821656 未加载

yalogin超过 11 年前

So what does it mean by facebook whitelisting a scraping service? Do they actively block scrapers?

评论 #6821643 未加载

pdfcollect超过 11 年前

Is there a way to replace this robots.txt with a null robots.txt? :)

评论 #6821603 未加载

bibstha超过 11 年前

What is a User Agent: Yeti?

评论 #6822826 未加载

decasteve超过 11 年前

Even Facebook's robots.txt has a hatred for my pseudo-anonymous browser settings. Facebook gives me this (for any page): "Sorry, something went wrong. We're working on getting this fixed as soon as we can."

评论 #6821610 未加载