The problem with all of this is that the reducto-scientific paradigm for understanding science has long been extended to communicating and teaching about science.<p>As the author does a really nice job explaining, and as seen in the Fisher quote, these articulations of heuristics and guidelines are often taken as closed form rules.<p>Look beyond statistics and you see it everywhere. Common ones include entrepreneurship and design...in both areas, experts' ways of thinking are often highly situated, highly metacognitive, and the actions they take are inherently inseparable from their thinking process. However, because of the academic drive towards objective/deterministic/observable phenomenon the research tends to report and attribute only the actions. The result is that those actions, rather than the underlying thinking processes, are valued and taught.<p>The result is simulations of expertise masquarading as knowledge. Its one thing when its students, but as you are seeing in psychology's 'replication crisis' (which, side note, is kind of a metaversion of its own critique) it can create real problems when surface level understanding is accepted and generalized as a normative 'truth' in a field. You see it in economics and business a lot...strive to appear scientific, but do so in ways that inherently betray the underlying structure of what you are studying. It comes from an underlying value in those communities, and society, that the only truth is objective truth.<p>If I have an experiment where I am screening 5 possible predictors and I get p values of .9 for 4 of them and .52 for 1...I would be an idiot not to pursue the 1. if I get 4 .49s and 1 .00000000001...same thing. Statistics is relative, literally.<p>[happy to provide citations...not sure anyone really cares]
The standard ritual for measuring significance in research seems to me to be some strange marriage of the ideas of Fisher, Neyman, and Pearson that I'm not sure any of them would have actually agreed with. I'd be interested to hear any historians of statistics or scientific methodology comment more on that angle or correct my misinterpretation if thats what it is.
Semi-off topic - as someone who does a lot of data analysis with sql but has never taken a statistics course - can anyone recommend any resources about where to learn about how to calculate / apply p-value, r-square etc?
p-values are very misunderstood and a lot more subtle than most people believe.<p>If you're interested in p-values, I wrote a post on them here with some counterintuitive examples (one of them shows how a lower p-value can sometimes increase your belief in the null-hypothesis).<p><a href="https://mindbowling.files.wordpress.com/2016/07/pvalues.pdf" rel="nofollow">https://mindbowling.files.wordpress.com/2016/07/pvalues.pdf</a>
What frightens me more is how rarely I see talk about decision theory and hypothesis testing in the (deep) machine learning community, it is as if people consider the classification output as sufficient evidence of recognition, just quote that max(p(class)) rather than the significance of the class given its classifier score.
Am I missing something?
Someone needs to reference the XKCD strip on jelly beans.<p><a href="https://xkcd.com/882/" rel="nofollow">https://xkcd.com/882/</a>
Scientists needs to start using a "training set" and a "test set".<p>If you have 2000 samples of data, you don't train your model on it and then say that's your success rate. You'll end up with conclusions that don't generalize.<p>Instead train on 1600 and measure your success on the remaining 400.<p>Similarly, don't look for statistical significance among your 2000 samples and conclude that's the result. Do it across 1600 and then validate it on the 400. If there is a real result there, it'll reproduce. It now makes your process robust to overfitting / param hacking.<p>You avoid the green jelly bean problem entirely.