>>"Unlike the Benchmarks Game, we aren’t requiring all the test programs for the same language to be identical. This is a controversial rule..."<<<p>Benchmarks Game programs for the same language are not required to be identical. Wilfred Hughes has been asked to correct that misstatement.<p>>>"It’s also not clear how representative the test programs are of typical performance of that language."<<<p>Without sampling programs "in the wild" how could anyone possibly claim that other programs were "representative"?<p>See <a href="http://research.microsoft.com/en-us/projects/jsmeter/" rel="nofollow">http://research.microsoft.com/en-us/projects/jsmeter/</a>
Wilfred, please correct your statement that the benchmarks game requires "all the test programs for the same language to be identical".<p>It isn't true. It wasn't true 4 years ago.<p>For sure, my preference was to show PyPy programs that also <i>worked</i> with CPython -- that made clear that optimizing for PyPy could make performance worse with CPython and <i>vice versa</i>.