Take the time to read the story (many comments don't reflect more than the headline), which is far more complex and interesting than the headline. For one thing, almost all studies' effects were reproduced, but they were generally weaker.<p>* Most importantly, from the Times: <i>Strictly on the basis of significance — a statistical measure of how likely it is that a result did not occur by chance — 35 of the studies held up, and 62 did not. (Three were excluded because their significance was not clear.) The overall “effect size,” a measure of the strength of a finding, dropped by about half across all of the studies. Yet very few of the redone studies contradicted the original ones; their results were simply weaker.</i><p>* Also: <i>The research team also measured whether the prestige of the original research group, rated by measures of expertise and academic affiliation, had any effect on the likelihood that its work stood up. It did not.</i><p>* And: <i>The only factor that did [affect the likelihood of successful reproduction] was the strength of the original effect — that is, the most robust findings tended to remain easily detectable, if not necessarily as strong.</i><p>* Finally: <i>The project’s authors write that, despite the painstaking effort to duplicate the original research, there could be differences in the design or context of the reproduced work that account for the different findings. Many of the original authors certainly agree.</i><p>* According to several experts, there is no reason to think the problems are confined to psychology, and it could be worse in other fields. The researchers chose psychology merely because that is their field of expertise.<p>* I haven't seen anything indicating the 100 studies are a representative sample of the population of published research, and at least one scientist raised this question.
As a marketer, I have long given up on psychology as a field. The practitioners are too wishy-washy, the experiments too prone to confirmation bias, and the results too unusable.<p>I feel behavioral economists cover many of the same subjects, but their work is much more interesting and scientific. The experiments are tighter and ask better questions. And they focus more on specific characteristics in decision making, and less on asking big questions.
Perhaps the biggest take away I found while studying psychology is that a lot of research in the field just couldn't be trusted until you vetted it. Political pressures are too great a corrupting factor. Words would be redefined. Conclusions overstated or over applied. This is on top of the already existing 'publish or perish' issue that impacts science as a whole.<p>The closer one was to neurology (like physiological psychology), the better it became. The closer one was to sociology (like IO psychology) or to a politically charged issue, the worse it became.<p>Also applies to psychiatry. I remember reading some of the papers published related to the DSM-V and at one point it looked like little more than peer reviewed version of two siblings fighting (though that was the worse case, not the average).<p>One big thing is to look at how the researchers defined words and look into how things translate when multiple languages were involved.
Here is the paper the Reproducability Project just released on Estimating the reproducibility of psychological science:
<a href="http://www.sciencemag.org/content/349/6251/aac4716" rel="nofollow">http://www.sciencemag.org/content/349/6251/aac4716</a>
Here's a nice, approachable treatment of some of the ways psych studies can go wrong; it also offers concrete suggestions for fixes:<p><pre><code> Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn
"False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant"
Psychological Science November 2011 22: 1359-1366,
</code></pre>
Free full-text (no paywall): <a href="http://pss.sagepub.com/content/22/11/1359" rel="nofollow">http://pss.sagepub.com/content/22/11/1359</a><p>(via: <a href="http://bahanonu.com/brain/#c20150315" rel="nofollow">http://bahanonu.com/brain/#c20150315</a> )
There has been a backlash against attempts at replication in psychology: <a href="http://phenomena.nationalgeographic.com/.../failed.../" rel="nofollow">http://phenomena.nationalgeographic.com/.../failed.../</a>
<a href="http://andrewgelman.com/2013/12/17/replication-backlash/" rel="nofollow">http://andrewgelman.com/2013/12/17/replication-backlash/</a>
and even a backlash to the backlash: <a href="http://phenomena.nationalgeographic.com/.../failed.../" rel="nofollow">http://phenomena.nationalgeographic.com/.../failed.../</a>
The replication effect sizes are almost all positive [1]. This study represents very strong validation that the published results were qualitatively true, in the main.<p>The regression to the mean effect is unsurprising and doesn't diminish this finding. Given the difficulty of performing this kind of research, this is a very positive result for the field.<p>[1]<a href="http://m.sciencemag.org/content/349/6251/aac4716/F1.expansion.html" rel="nofollow">http://m.sciencemag.org/content/349/6251/aac4716/F1.expansio...</a>
For anyone interested in more information on this topic, the book "Psychology Gone Wrong" by Tomasz Witkowski and Maciej Zatonski is a very in depth look into the problems in this field.
This is how I evaluate studies, "are the data and code made available?" It's a simple request and this study is the first I've ever seen offer both. Bravo!