If nothing else, this succinctly describes how to vet decision processes based on statistical tests.<p>I ran into a situation within my own organization where costly decisions were being made on variations in statistical test results that I suspected were local minima/maxima in natural variance. Replaying the decision process against the same target multiple times provided strong evidence towards changing how we did that.<p>It wasn't enough to just calculate the estimated population variance, etc., because explaining the significance of that was essentially a TL;DR problem to anyone whose opinion counted. It also required significantly better stats skills than I had, since adequately explaining the issue generally requires more expertise than recognizing it in the first place.<p>However, empirically demonstrating that the decision process was flipflopping on the same available information was extremely compelling.