"...Yahoo trumpeted data showing their results were better than Google's. Well, the Google team was quite amazed, because their data showed just the opposite: their results were better than Yahoo's... It turns out that Yahoo's benchmark contained queries drawn from Yahoo search logs, and Google's benchmark likewise contained queries drawn from Google search logs. The Yahoo ranking algorithm performed better on the Yahoo benchmark and the Google algorithm performed better on the Google benchmark."<p>Really? That's hugely important.
This is a fundamental lesson in quality testing. Nothing new, but always good to be remind of. Plus, it will be more interesting to study the human biases that often mislead us, and create a paradigm that minimize such biases.