> > Each experiment quotes a likelihood of very close to “5 sigma,” meaning the likelihood that the events were produced by chance is less than one in 3.5 million.<p>> These words come from Lawrence Krauss, a highly respected theoretical physicist, and they are wrong.<p>I don't get what is wrong with it, "likelihood that the events were produced by chance" sounds exactly like "P(D|H_0)" i.e., the probability that we see given data (or more extreme) assuming null hypothesis (no systematic error produced the data, just random chance).<p>Numbers also correct one-in-3.5 million is what we expect from "5 sigma" (one-tail):<p><pre><code> import statistics
print(f"{1 - statistics.NormalDist().cdf(5):.3g}, {1/3.5e6:.3g}")
# 2.87e-07, 2.86e-07</code></pre>
I always bring this up in discussions with social scientists and economists who maintain 99% or 95% significance level as the standard above which results from their studies get accepted. It's ridiculous; we can expect 1 or 5 in a hundred studies to be wrong just from statistics.