homepage pop-up: "At approximately noon on Sunday August 5th, Craigslist instructed all general search engines to stop indexing CL postings -- effectively blocking 3taps and other 3rd party use of that data from these public domain sources. We are sorry that CL has chosen this course of action and are exploring options to restore service but may be down for an extended period of time unless we or CL change practices. As soon as we know more, we will share it here and on our Twitter account."
I don't think this is accurate. As far as I can tell, there is nothing in CL's robots.txt, meta tags, or response headers that prevents Google from indexing them. Further, requesting a CL post with the Googlebot user agent yields the same content. This only leaves the possibility that they are excluding Google via specific IP blocks, which seems unlikely. Is there something I'm missing?
<a href="http://blog.sfgate.com/techchron/2012/08/10/craigslist-backs-off-exclusive-rights-to-ads/" rel="nofollow">http://blog.sfgate.com/techchron/2012/08/10/craigslist-backs...</a><p>"One data harvester, 3taps, said earlier this week that Craigslist had blocked search engines such as Google from including Craigslist pages in search results. But that report was inaccurate.<p>3taps’ product and quality assurance leader, Meg Nakamura, acknowledged Wednesday in a chat with The Chronicle that something fishy was taking place, but developers there haven’t fully figured out what’s going on."
<a href="http://www.sfgate.com/technology/businessinsider/article/Craigslist-Is-Definitely-Blocking-Search-3769297.php" rel="nofollow">http://www.sfgate.com/technology/businessinsider/article/Cra...</a><p>Not sure I agree with most the conclusions drawn in that article.<p>The article does say that "sure enough, Google displays recent listings from Craigslist right now," which does seem to be true for me, too, when I try.
<a href="https://twitter.com/markmilian/statuses/233015694432813057" rel="nofollow">https://twitter.com/markmilian/statuses/233015694432813057</a><p>Mark Milian @markmilian 7 Aug<p>Contradicting earlier statement, 3Taps spokeswoman emails to say, "Craigslist is still allowing indexing of pages." Still nothing from CL PR
Actually the part about search engines doesn't seem to be true... I just performed searches using Google, Yahoo, and Bing and got links to CL postings that were made within the last hour.