Search engine relevance - an empirical test

Yes, it is true that Live, Yahoo and Google are pretty close in terms of relevancy. People think of Google being the best, but the fact is that other engines are catching up. There are 200 search engines out there and most of us only know about the 4 big ones. Back in 1997, everyone used Yahoo or MSN and thought they were the best. Same applies to Google in 2008. There will be other players that will take over in the next few years.

I think the test is a good idea but the 'relevance raters' are only looking at the listing results - not the actual page. The relevance of the actual page is far more important to the user experience than the listing text. A clever 'made for adsense' site could scam the listing but obviously not the actual content.

The more important question is, for the prices they are paying for labor over at Mechanical Turk were the participants sufficiently literate in the English language.