Science has lost its way, at a big cost to humanity

<i>&quot;The demand for sexy results, combined with indifferent follow-up, means that billions of dollars in worldwide resources devoted to finding and developing remedies for the diseases that afflict us all is being thrown down a rathole. NIH and the rest of the scientific community are just now waking up to the realization that science has lost its way, and it may take years to get back on the right path.</i>&quot;<p>The institution of science is undergoing a catastrophic decline. The reason behind this is simple: it is no longer a growing economy. Public funding for science is frozen or being cut, private R&amp;D labs are shuttering their doors, and companies are increasingly concerned with quarterly results at the expense of long term research.<p>And why should it be otherwise? Science has never payed off as a logical financial investment. It is the riskiest of gambles by definition, requiring inordinate expenditures of time and resources in the present for a chance at some distant breakthrough decades or even centuries in the future. Institutional science is not an economically sound choice in the best of times, let alone during the current span of never-ending recessions.<p>The truth is, science is a creative pursuit much like the arts. Like the creation of literary masterpieces or profound paintings, it has never made economic sense in the present. Only afterwards, once the impact can be seen, do we understand its significance. And that is why it will always be worth pursuing.<p>The reality is that, increasingly, we live in a society that does not understand this philosophy of life. People only care about how they will survive tomorrow, and who can blame them, as the world economy gets ever more competitive and cut-throat.<p>Increasingly, it has become clear that our society does not reflect one designed with its own best interests at heart. Why this is, how it happened, and how we can change it, will be the greatest challenge of our lifetimes.

There&#x27;s nothing wrong with criticizing science, but the reaction to <i>The Economist</i> article -- which itself was a bit too breathless for comfort -- is heading rapidly into tiny-green-football-linkbait territory.<p>The scientific funding and publication system has problems that deserve scrutiny, but <i>science</i> itself is far more rational than nearly any large, human-maintained system I can think of.<p>When we resort to hyperbole like <i>&quot;science has lost it&#x27;s way&quot;</i>, we give a group of vocal, clueless idiots more power to undermine the most consistently productive engine for progress that humanity has ever devised. So let&#x27;s talk rationally about the problems, but don&#x27;t throw the baby out with the bathwater.

First of all, criticizing science is like criticizing democracy. Yes, it&#x27;s flawed, but still far better than anything else we&#x27;ve tried so far!<p>If you look at the typical application for funding, you&#x27;ll see questions that basically prod you to explain why your research&#x2F;students&#x2F;etc. are exceptional&#x2F;revolutionary&#x2F;ground-breaking. Everything must look like a Nobel prize waiting to happen if you&#x27;re going to have a chance at beating out everyone else making the same application (and exaggerations). It&#x27;s utterly ridiculous! It&#x27;s as if thousands of guitarists were auditioning at the same time and, in an effort to be heard, each has cranked their amp to 11. The result is a cacophony where even each individual sounds awful because of the distortion. If everyone dialed it down to 5 things would be bearable, but there&#x27;s always someone willing to nudge it up to 6 or 7...<p>A nice long list of high profile publications is great to hype when your amp is set to 11. If you have published many papers in high impact factor journals (again, often by inflating the significance of your work), you must be worth funding!<p>Perhaps scientific funding needs to be awarded in a manner that is more... scientific. Heck, perhaps funding agencies should reserve a certain percentage of their funding specifically for reproduction of results. Currently, if you apply for a grant to check other peoples work, people doing original research will win absolutely every single time. Unfortunately, the preference for original research goes right to the very top of governments. Politicians want brilliant nobel price winners, not competent fact-checkers.

One idea to improve the state of things is to require graduate students to verify some number of external studies. In addition to helping with the problem of not enough review, it would make an excellent practical test for doctoral candidates.<p>It wouldn&#x27;t work for every field and area, but it could work for a significant subset of research.

I’m a scientist and I agree with this article. Fact is that the way we incentivize science is what is causing these issues (that and tenure, but that is a longer post).<p>Science, at least in biology, is just like any business. Both are motivated by $.<p>The good news is that open science and the impact that the internets is having on science can help this problem. In my opinion, transparency in science will fix many of these issues.

A nod should be thrown out here to John Ioannidis, who has been banging this drum for a while.<p>1. <a href="http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/308269/" rel="nofollow">http:&#x2F;&#x2F;www.theatlantic.com&#x2F;magazine&#x2F;archive&#x2F;2010&#x2F;11&#x2F;lies-dam...</a><p>2. <a href="http://marginalrevolution.com/marginalrevolution/2005/09/why_most_publis.html" rel="nofollow">http:&#x2F;&#x2F;marginalrevolution.com&#x2F;marginalrevolution&#x2F;2005&#x2F;09&#x2F;why...</a><p>Marginal Revolution also throws a tip of the hat to Brad deLong and Kevin Lang&#x27;s paper condemning econ journals for refusing to print any piece that fails to reject the null hypothesis: <a href="http://www.jstor.org/discover/10.2307/2138833?uid=3739936&amp;uid=2&amp;uid=4&amp;uid=3739256&amp;sid=21102824481747" rel="nofollow">http:&#x2F;&#x2F;www.jstor.org&#x2F;discover&#x2F;10.2307&#x2F;2138833?uid=3739936&amp;ui...</a><p>Speaking of econ, this feels like a market failure to me. I&#x27;d be in favor of redirecting some portion of government research dollars towards an independent &quot;validation&quot; shop staffed by scientists who attempt to independently replicate submitted findings, and vow to write up all results (even detailing events that lead to the interruption or cancellation of an experiment). Findings that cannot be replicated by the validation shop should be viewed with extreme skepticism. Researchers would quickly learn not to fudge the results, and find more effective ways to control their own unintentional biases.<p>It wouldn&#x27;t have to be government. It could be useful as a nonprofit, I just don&#x27;t think it&#x27;d be sexy enough for anyone to support, despite the sort of urgent necessity of something like this.

&quot;Because of science - not religion or politics - even people like you and me can have possessions that only a hundred years ago kings would have gone to war to own. Scientific method should not be take lightly.<p>The walls of the ivory tower of science collapsed when bureaucrats realized that there were jobs to be had and money to be made in the administration and promotion of science. Governments began making big investments just prior to World War II...<p>Science was going to determine the balance of power in the postwar world. Governments went into the science business big time.<p>Scientists became administrators of programs that had a mission. Probably the most important scientific development of the twentieth century is that economics replaced curiosity as the driving force behind research...<p>James Buchanan noted thirty years ago - and he is still correct - that as a rule, there is no vested interest in seeing a fair evaluation of a public scientific issue. Very little experimental verification has been done to support important societal issues in the closing years of this century...<p>People believe these things...because they have faith.&quot;<p>From Kary Mullis, the Nobel Prize in Chemistry winner (and the genius inventor of PCR) in an excellent essay in his book &quot;Dancing Naked in the Mind Field&quot;.

Real science depends on room to fail, but starting in middle school science fairs, it&#x27;s clear that negative results and &quot;failed&quot; experiments aren&#x27;t what the teachers&#x2F;judges are looking for.<p>I made it through to the state science fair in 8th grade. It was based around magnetism, and after months of work, it turned out my tests just weren&#x27;t sensitive enough to measure any difference in any of the electromagnets I built.<p>When I mentioned this to my teachers, they encouraged me to <i>fix</i> the results with a wink and a nod. Sure I could have turned in a &quot;failed&quot; project and maybe got a B, but I was an A student and there was no room for failure.<p>I&#x27;d love to see some data on how many high level science fair projects are faked each year.

This is not only in medical science, but apparently archaeogy too. (Anecdote alert) A very close friend took part in a dig by a very well known professor. Artifacts that were found that disproved the professor&#x27;s theories were destroyed before my friend&#x27;s eyes.<p>It really makes you wonder what percentage of what we &quot;know&quot; is true.

There are a lot of issues flying around that are being inappropriately mixed up for all sorts of political purposes.<p>The Begley &#x27;study&#x27; is impossible to assess, because they didn&#x27;t report what the studies were, nor any of their methods, or anything. It&#x27;s BS and hearsay, not science. Moreover, according to the Begley article itself, &quot;The term &#x27;non-reproduced&#x27; was assigned on the basis of findings not being sufficiently robust to drive a drug-development programme.&quot;<p>Nobody said that the purpose of every single scientific paper was to enable Amgen to go start a drug-development program.<p>There are many problems with the science funding situation, the glamour pub game, excess hype, funding getting sucked up by mega-projects, lack of open-access, inability to publish negative results, etc, etc, etc, but in general, it is not true that scientists are making up sexy results to get them into Nature and Science.

This is just way too true. I know someone (anecdote alert) learning at a really prestigious university in a medical field and she told me multiple times that they intentionally cheat the results to match the expected output.<p>By cheating I mean... flat out lying. I don&#x27;t know the implications of this (how far misinformation can get) but it seems like a wrong culture and attitude, especially for science.

&quot;lost its way&quot; suggests that science was once firmly on a sure path of rigorously verified studies, never a thrice-checked statement assumed. That was never the case.<p>Acknowledging, attempting to quantify, and then (some institutions) attempting to fix systemic issues in the peer review system is not an emerging crisis. We have to fix incentives, but we aren&#x27;t about to see some fundamental tenets about to be overthrown here.<p>It isn&#x27;t clear what the &quot;big cost&quot; is referring to. Certainly money has been spent on poorly-founded studies with fundamentally inconclusive results. If it instead refers to opportunity cost, fortunately we have the entire future of humankind to pick up what we might have figured out earlier.

While I certainly agree with much of the factual content presented both in the article and in the comments, I think that science already has a lot of self-correction mechanisms built in. None are perfect individually, but the big, messy system has a lot of redundancy built in. It&#x27;s just not always so visible to journalists or science writers, who don&#x27;t hang around the scene for the years that it often takes for science to find its way again, so to speak.<p>For example, many of these high-profile, possibly erroneous (or occasionally fraudulent, it seems) Nature or Science articles are high-profile because they seek to address a contentious or long-standing problem in the field. When this happens, there are typically existing alternate hypotheses. It&#x27;s much easier to get papers published or grants funded by seeking to test competing hypotheses than to simply try to verify an isolated study. It can also be easier to find weaknesses in an individual study by testing it in a different way, or against other models, or whatever, than by simply trying to reproduce it. Often, a single study might be impossible to directly replicate, or the underlying flaws may not be apparent until the problem is approached from a different angle.<p>Granted, this can take a couple years or even decades, but falsehoods (intentional or not) tend to become more apparent as their context becomes more clear.

Science Exchange is on it: <a href="https://www.scienceexchange.com/reproducibility" rel="nofollow">https:&#x2F;&#x2F;www.scienceexchange.com&#x2F;reproducibility</a>

Science is working exactly the way it has always worked.<p>Most papers have always been flawed, wrong, or not reproducible. There has always been pressure to publish--going back even to Newton&#x27;s battles with Hooke over gravity, or Darwin&#x27;s rush to publish <i>On the Origin of Species</i> before Wallace.<p>What has changed are the cultural expectations. Culturally, we&#x27;ve become spoiled by physics. We&#x27;re used to the precision, speed, and accuracy of physics and engineering. Moore&#x27;s law, the iPhone, incredible bridges, the 787 and 380 airplanes--they all just work, safely and reliably.<p>Note that the reproduction problems are most prevalent in chemistry, biology, medicine, etc. These are areas of science that are far more complex, and about which we know far less, than physics. It will take a long time, and a lot of failed research, to even start to approach that level of knowledge. Given the complexity, it might be impossible.

What I&#x27;m reading in the article is claims that science <i>funding</i> has lost its way, and is rewarding exactly the wrong actions with money and prestige.

The title should read: &quot;A subset of the Scientific Community has lost it&#x27;s integrity, at a not so easily quantifiable cost to humanity&quot;

If you are a biologists and you want to keep your lab going, and you want to have RA-ships for your graduate students, you need to get grants. You aren&#x27;t going to get grants unless you are cranking out publications. The days when as a biologist, you could work on a problem for several years, being careful, checking your work before you publish... those days are over. I&#x27;m confident the system will right itself eventually, hopefully in my lifetime.

It seems like the solution to this is fairly simple. Use some statistical or machine learning method to figure out the probability that a certain thing is true using the information we know about it, like what journal it was published in, the results of replications, maybe even stuff like how crazy the result seems or the experience&#x2F;reputation of the scientists, etc. There is a ton of data to work with, on top of the actual data itself.<p>You could predict with decent accuracy how probable a study is to turn out to be true or false. Then you can use that information to decide whether it would be worthwhile to do more studies.

The PubMed Commons initiative[1] by the National Institutes of Health, mentioned in the article kindly submitted here, is a start at addressing the important problems described in the article. One critique[2] of the PubMed Commons effort says that that is a step in the right direction, but includes too few researchers so far. A blog post on PubMed Commons[3] explains a rationale for limiting the number of scientists who can comment on previous research at first, until the system develops more.<p>[1] <a href="http://www.ncbi.nlm.nih.gov/pubmedcommons/" rel="nofollow">http:&#x2F;&#x2F;www.ncbi.nlm.nih.gov&#x2F;pubmedcommons&#x2F;</a><p>[2] <a href="http://retractionwatch.wordpress.com/2013/10/22/pubmed-now-allows-comments-on-abstracts-but-only-by-a-select-few/" rel="nofollow">http:&#x2F;&#x2F;retractionwatch.wordpress.com&#x2F;2013&#x2F;10&#x2F;22&#x2F;pubmed-now-a...</a><p>[3] <a href="http://www-stat.stanford.edu/~tibs/PubMedCommons.html" rel="nofollow">http:&#x2F;&#x2F;www-stat.stanford.edu&#x2F;~tibs&#x2F;PubMedCommons.html</a><p>USING MY EDIT WINDOW:<p>Some of the other comments mention studies with data that are just plain made up. Fortunately, most human beings err systematically when they make data up, making it look too good to be true. So an astute statistician who examines a published paper can (as some have done) detect made-up data just by analyzing what data are reported in a paper. A researcher who does this a lot to find made-up data in psychology is Uri Simonsohn, who publishes papers about his methods and how other scientists can apply the same statistical tests to find made-up data.<p><a href="http://opim.wharton.upenn.edu/~uws/" rel="nofollow">http:&#x2F;&#x2F;opim.wharton.upenn.edu&#x2F;~uws&#x2F;</a><p>From Jelte Wicherts writing in Frontiers of Computational Neuroscience (an open-access journal) comes a set of general suggestions<p>Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389&#x2F;fncom.2012.00020<p><a href="http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00020/full" rel="nofollow">http:&#x2F;&#x2F;www.frontiersin.org&#x2F;Computational_Neuroscience&#x2F;10.338...</a><p>on how to make the peer-review process in scientific publishing more reliable. Wicherts does a lot of research on this issue to try to reduce the number of dubious publications in his main discipline, the psychology of human intelligence.<p>&quot;With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses.&quot;

There is no reproducibility crisis in science.<p>1. Reproducing work is a waste of resources. Use the money and researcher hours to develop tools that are more reliable, cheaper, and easier to use. A major reason why no one reproduces experiments is that the initial work was very difficult. Let&#x27;s invest in technologies that make science easier. Reproduction (of experiments) should be done in high school biology classes.<p>2. Technical reproduction is rarely done, but conceptual reproduction is common. Findings in the literature become incorporated into disparate subsequent hypotheses tested by many other labs. If something doesn&#x27;t add up, this will often increase the impact of the paper and eventually be addressed through experiments to resolve different models of the phenomenon.<p>3. There is no widespread fraud in science. Your academic career rests on your integrity. When I publish a paper, I do my damn best to make sure it is accurate. My reputation relies on it. When scientists continue to publish results that are false or fraudulent, they become discredited within the community. All graduate students in the life sciences are required to take a class on the ethics of science.<p>4. Publishing is a bitch and a source of real rot within the community. Fortunately, many researchers and academics recognize this problem and are addressing it. Look at the new journal eLife, or open access journals, or the increasing interest in arXiv.org (moving to a publishing model closer to that found in math and physics, which appear to be healthier research communities than life sciences). As experiments become more technically advanced, the expectation for methods sections have increased, not decreased.<p>5. People want to commercialize scientific findings that are relatively new - it&#x27;s obvious that is risky! Why put the burden back on (under-funded) scientists? Drug companies are the ones that would benefit financially. Or wait until the phenomenon is better understood. Notice they&#x27;re talking about drugs for complex diseases like cancer, metabolic disorder, etc., not Mendelian diseases. It&#x27;s as if people complained that they couldn&#x27;t get their lasers to work using a 1917 understanding of the physics of light. But Einstein demonstrated the fundamentals! Why did it take 40 years to make it work in practice?

Someone do a sentiment plot with &quot;goodness&quot; on the Y-axis and years ago relative to writing no the X-axis. I won&#x27;t be surprised if there&#x27;s a positive correlation. Successes, new challenges, and shortcomings become apparent. Whatever worked looks like it was good principle in hindsight. Whatever hasn&#x27;t panned out due to new challenges looks terrible. Cherry picking in order to build a case that allows one to write authoritatively doesn&#x27;t make anyone a saint or cultural leader.<p>Therefore, when I see an article like this with such a broad, generalizing headline, I just think it&#x27;s click-bait. Lost it&#x27;s way? I&#x27;ve read some absolutely terrible papers. &quot;Theory of the Origin, Evolution, and Nature of Life&quot; by Erik Andrulis is an excellent example of such unfathomably speculative garbage. I&#x27;ve also read a huge number of well-done papers on topics in aerospace engineering and materials science. It&#x27;s always on the reader to re-produce experiments if they depend on the result, to understand the paper correctly etc. This is what my professors did. If part of the community is circle-jerking, let evolution run its course. We used to treat Aristotle as canon in the western world. Obviously things get better over time.<p>Skimmed article. Old news. The fact that someone is raising the flag, saying &quot;there&#x27;s a lot of low-hanging fruit to use to establish yourself as a more accurate researcher,&quot; just means we will see more of such review activity, making the title seem inaccurate. You never know when you can free yourself up an adjunct professor position in exactly your preferred field of research.

Especially in medical studies, where you&#x27;ve often got cases of n=40 or similar (even in later stages!), this is a huge issue. In contrast, just think of the size of n you need in physics to be taken seriously!<p>The major reason for that is, however, that most people in the medical &amp; biological area are rather lacking a profound mathematical education. There are cases where papers get rejected because they are too mathematical.

In the UK, there is the widely-derided &quot;Research Assessment Exercise&quot;: <a href="http://en.wikipedia.org/wiki/Research_Assessment_Exercise#Criticism" rel="nofollow">http:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Research_Assessment_Exercise#Cr...</a><p>There is also the criterion of &quot;impact factor&quot;, for papers and publications. It&#x27;s very similar to a &quot;Karma&quot; system as used by HN, Reddit, etc., and it has many of the same problems. Imagine a system where you have to choose between doing research that might be vital but probably won&#x27;t, versus something safe and predictable that ensures that you get paid next year.<p>The problem is not so much science as the <i>management</i> and <i>funding</i> of science, which have been infected by the same managerialism that causes so many problems in big government and corporate projects.

Here&#x27;s a case study of mass-media attention-seeking through pseudo-science: an evaluation of behavior on sinking ships, coincidentally issued five days before the centennial of the Titanic&#x27;s sinking. POS, but well, ok, surely serious scientists wouldn&#x27;t take such work seriously. The Proceedings of the National Academy of Sciences of the United States of America (PNAS) received the paper for review on May 2, 2012, approved it on June 29, 2012, and published it on July 30, 2012. Science has lost its way.... For details, see <a href="http://purplemotes.net/2012/04/22/deadly-sex-discrimination-in-titanic-chivalry-myth-reporting/" rel="nofollow">http:&#x2F;&#x2F;purplemotes.net&#x2F;2012&#x2F;04&#x2F;22&#x2F;deadly-sex-discrimination-...</a>

While I would say that the drop in rigour is worse in Medicine than Physics, it is clearly still present even there.<p>The way funding works, in particular, means people publish stuff-that-will-get-references with a similar attitude to web start-ups iterating their code, (by which I mean too damn fast and not listening to the peer reviews.)<p>I&#x27;d love to see this change, but I don&#x27;t know how central agencies can easily&#x2F;affordably work out which research(ers) to fund and which to cut. As others have said, by definition we don&#x27;t know what work was useful&#x2F;valid&#x2F;critical until many years later.

Why do authors and publishers of articles like this, which invariably turn out to be about medical knowledge&#x2F;studies&#x2F;research, resort to using the misleading, incredibly broad word &quot;Science&quot; in their titles? All of &quot;Science&quot; has lost its way? Really? Phyics? Chemistry? How about this for a headline: &quot;Journalism has lost its way&quot;? The article could then simply be a list of all the sensationalist, purposely misleading crap that&#x27;s published in major publications. Long list.

This will continue because the tendency to make excuses for it is directly proportionate to the ability to do anything about it, same as with most&#x2F;all social ills.

The article refers to a move several years ago by one biotech company, Amgen, to attempt to validate the results of some well-known studies.<p>Where can we find the list of these studies?

Reproduction of results sounds like a good area of AI and automation. Thankless toiling, not very gratifying work, largely mechanical (not creative).<p>It would be very cool if someone came up with a unit test framework for various fields of science. Then we could make reproduction unit tests a requirement of publishing, so anyone with the proper equipment&#x2F;framework could sync and run the tests themselves.

In the spirit of inquiry, I&#x27;m waiting for the other half of this story. These articles seem sensationalist. So what&#x27;s the catch? Does it have to do with studies being pre-clinical and more likely to be wrong? Or is it that they&#x27;re being held to a different level of scrutiny? Or the evidence shows correlation, but not statistical significance?<p>Or are things really this bad?

Does this impact venues like HN and Wikipedia that value citations substantially more than common sense?

Why not allow comments from everyone? We all know that closed clubs lead to politics. Many life scientists are extremely allergic to feedback, and they go to great lengths to avoid scrutinizing their own or others&#x27; results.

When you make publishing a number of papers mandatory and tie it to tenure, it is inevitable that quality will suffer. The need is to rethink quantitative metrics and move to a more quality oriented one.

Society has lost its way.

I would highly recommend that anyone with further interest check out the book, &quot;The Cult of Statistical Significance&quot; It&#x27;s eye opening.

Science (management) has lost its way, at a big cost to humanity

climate change.