Measurement bias can have a huge effect, even when you think you are doing 'good' benchmarking.<p>See the paper 'How To Produce Wrong Data Without Doing Anything Obviously Wrong!' [1]. They just change the size of environment variables and the link order of files, and get wildly different performance results. Most people just assume if they close all other running programs they'll have a fair benchmark, but there is so much more to consider.<p>[1] - <a href="http://www-plan.cs.colorado.edu/diwan/asplos09.pdf" rel="nofollow">http://www-plan.cs.colorado.edu/diwan/asplos09.pdf</a>