I'm setting up a new site and noticed that within seconds of starting the server I was getting hits in the log like:<p><pre><code> 67.195.112.231 - - [2011-06-17 18:49:37] "GET /SlurpConfirm404/starsong/pro-road.htm HTTP/1.0" 404 18 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
67.195.115.174 - - [2011-06-17 16:53:30] "GET /SlurpConfirm404/drugstore.htm HTTP/1.0" 404 18 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
</code></pre>
My initial thought was that I was getting crawled by a spam bot masquerading as a Yahoo crawler, but after a bit of googling I found a couple of blogs guessing on the nature of the strange requests.<p>My best guess is that they're using the above test to check if your server is gladly responding even the most obscure request, thus making it look like a spider trap/content farm.