When my wife was working on her PhD in linguistics she ran across a highly referenced source that she could not find. She followed the chain of reference and reached out to the author who cited the source the first time. He replied that this citation was a typo in the original article and give her the correct source reference. He said that he was chagrined to have seen this typo cited in scores of papers with apparently no one bothering to actually look it up.
My first serious contact with scientific publishing was when I worked on master's thesis in bioinformatics on some very specific problem. I discovered that papers had systemic errors in input data, often statistics was not applied correctly, the full source code mostly not available, yet the papers built on top of each other, presenting results.<p>The scientists I worked with seemed to know about the situation, but personally, I lost the very strong belief in natural science results. Turns out, there was no rigorous fact and results checking culture, as in mathematics.<p>It seems to me, it is mostly the lay people who treat published scientific results as some proven ground truth, the scientists themselves are more relaxed about it. Science may be in some ways about telling interesting stories in a convincing way.
The same is true in computer science. Everyone cites the SHARD paper when discussing database design. But no modern researcher has read it. It does not appear in any archive.<p>I spoke to several academics who had mentioned it, none had read the original - they all relied on someone else's referencing of it.<p>See <a href="https://shkspr.mobi/blog/2021/06/where-is-the-original-overview-of-shard-paper/" rel="nofollow">https://shkspr.mobi/blog/2021/06/where-is-the-original-overv...</a> for more details.
Not sure if it's in the exact same category, but one of the most devastating examples was a one paragraph entry in Letters to the Editor, published in 1980 in the New England Journal of Medicine, that I first heard about in the excellent Dopesick dramatisation. This unverified comment from a reader became legitimised as it morphed to a Journal citation, subsequently referenced in hundreds of first-gen papers, and who knows how many indirect references.<p>It effectively opened the door to Purdue's marketing department.<p>"In conclusion, we found that a five-sentence letter published in the Journal in 1980 was heavily and uncritically cited as evidence that addiction was rare with long-term opioid therapy.<p>"We believe that this citation pattern contributed to the North American opioid crisis by helping to shape a narrative that allayed prescribers’ concerns about the risk of addiction associated with long-term opioid therapy."<p><a href="https://www.nejm.org/doi/full/10.1056/NEJMc1700150" rel="nofollow">https://www.nejm.org/doi/full/10.1056/NEJMc1700150</a>
I thought that this was going to have stories like when taking an exam in a large lecture hall, one student finishes late, and the professor says, "Sorry, you can't turn that in now." And the student goes, "Do you know who I am?" where the professor replies, "No, not at all." Then the student goes, "Good," and shoves his paper in the middle of the pile of exams.
HN and similar forums are particularly prone to this behavior. We see two messages:<p>* Message A: X is true<p>* Message B: Message A is false<p>Based only on that information (which often is all we have), there is no reason to believe message B rather than message A, but people seem to love a 'debunking' (it seems to make them feel smart).<p>I seem to remember that it matches a cognitive bias, such as believing the more recent message. Or, maybe we just fall for whoever acts more confident - a literal con game: The person communicating B is claiming to know more than the person communicating A, and for some reason we believe them.
A close friend of mine was doing her PhD in nutrition science (or something like this) and asked me to review her work from a math (especially statistics) aspect.<p>I started to read it but it was so bad that I could not go on. I told her to get someone from her field because I am too traumatized to read that and if she takes my input into account she may well need to rewrite the whole thing.<p>I was at her defense and some people asked questions about the statistical part, but the level was "how do you calculate the average".<p>This worries me a bit because as far as the topic is concerned, statistical mistakes about Middle Ages in France are sad but that's it - but mistake sin nutrition or pharma can be worrisome. In her case that was quite esoteric so no risk for anyone but generally speaking I now do not trust any numbers I cannot analyze myself.
This was a really good read! It strikes at the heart of some frustration I feel when people get smug about "the science" as if they have checked anything for themselves. Seriously: be less confident.
I'm still only partway through the article, but wonder how much of this problem is due to things like introductions which cite prior knowledge. If non-review papers only reported the data they actually collected they'd be a lot more boring, but also less prone to urban legend transmission.<p>> How should I refer to my source? If I want to include this sentence in an academic publication, what should I place after my sentence?<p>Why would you include it in an academic publication? Just cite a correct source for the correct data. And maybe, at most, cite the original incorrect source and state that it was incorrect.
The well cited “Conways Law” in technology falls into this category. I can barely remember a week going past in the last 3 years without someone quoting it. As part of my PhD research, I’ve read the paper deeply, and the idea that it proves the companies ship their organisational structures is such a misuse of the research. I wonder how many people have ever read the original
The spinach story reminds me a lot on the false recommendation of siphash for hash table DDOS prevention. <a href="https://github.com/rurban/smhasher#security">https://github.com/rurban/smhasher#security</a><p>The authors came up in their widely cited paper with a proper solution to spread the random hash seed into the inner loop, vastly enhancing its security by avoiding trivial hash collision attacks. But a secure, slow hash function can never prevent from normal hash seed attacks, when the random seed is known somehow. esp. with dynamic languages it's trivial to get the seed externally.<p>Other trivial countermeasures must be used then, which also don't make hash tables 10x slower, keeping them practical.
So is spinach a good source of iron or not? I mean when taking the whole absorption thing into the account too. Sorry, I'm in a hurry and I have no time to read the article carefully and I must contact my mom.
When I wrote class term papers in the 1970s I spent days in the massive book stacks of Harvard libraries. I suspect many urban legends about love-making, deceased souls, and crimes in the stacks may true. Like in the movie The Paper Chase.<p>I dont know if Harvard Libraries has warehoused most their physical books like many other college libraries. One has to order up old books then.
That's a cute story (read to the end for the twists). I really came to hate this problem when reading COVID research.<p>Primary case for the prosecution is a paper like Flaxman et al, "Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe" [1]. It claims that lockdowns saved ~3 million lives. Amongst other problems this paper has: it is built on circular logic [2] and an over-fitted model [3], has numerous other statistical problems [4], required an enormous country-specific fudge factor and hiding data points to cover up the way Sweden's results disproved their claims, and their model always claimed that the last NPI was the most effective regardless of what it was.<p>The cherry on top is that it disclaims its own results, literally admitting in the text that their counterfactual model is <i>"illustrative only"</i> and that <i>"in reality, even in the absence of government interventions we would expect Rt to decrease and therefore would overestimate deaths in the no-intervention model"</i> i.e. they already knew their conclusions were wrong when they wrote it. (This rather important caveat didn't appear in their press release about the paper, of course [5]).<p>Yet according to Google Scholar this paper has been cited nearly 3000 times in the ~three years since its publication. It gets cited several times <i>per day</i>. I've watched the citation count go up with morbid fascination. What are people citing it for?<p>If we do a reverse citation search and check some papers, we see immediately that it's being cited for a wide variety of almost random statements, none of which it actually supports:<p>1. "Health-care workers, seniors and those with underlying health conditions are at particularly high risk" [6] [8]. This claim appears with identical wording in two different papers, but Flaxman et al don't present any data on this or even reference risk stratification by job as far as I can tell. It's certainly not the focus of the paper.<p>2. "However, because most countries have implemented multiple infection control measures, it is difficult to determine the relative benefit of each" [7]. Flaxman et al claim it's easy to determine the benefit of each and that they did so.<p>3. "health agencies have long relied on predictive models to estimate future trends and to assess the potential effectiveness of various disease control methods" [9]. The paper doesn't show anything about the history of health agency decision making.<p>etc. Even when citations characterize its claims correctly, they are just taking its assertions at face value without realizing that the paper's methodology is circular and even the authors don't believe their own numbers. Of what use are citations in this environment? These aren't cherry picked examples, they're literally just randomly selected by when I happened to search Scholar. Even so, <i>most</i> of the citations are wrong.<p>[1] <a href="https://www.nature.com/articles/s41586-020-2405-7" rel="nofollow">https://www.nature.com/articles/s41586-020-2405-7</a><p>[2] <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7674856/" rel="nofollow">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7674856/</a><p>[3] <a href="https://www.nature.com/articles/s41586-020-3025-y" rel="nofollow">https://www.nature.com/articles/s41586-020-3025-y</a><p>[4] <a href="https://nicholaslewis.org/did-lockdowns-really-save-3-million-covid-19-deaths-as-flaxman-et-al-claim" rel="nofollow">https://nicholaslewis.org/did-lockdowns-really-save-3-millio...</a><p>[5] <a href="https://www.imperial.ac.uk/news/198074/lockdown-school-closures-europe-have-prevented/" rel="nofollow">https://www.imperial.ac.uk/news/198074/lockdown-school-closu...</a><p>[6] <a href="https://www.nature.com/articles/s41577-020-00434-6" rel="nofollow">https://www.nature.com/articles/s41577-020-00434-6</a><p>[7] <a href="https://jamanetwork.com/journals/jama/article-abstract/2768391" rel="nofollow">https://jamanetwork.com/journals/jama/article-abstract/27683...</a><p>[8] <a href="https://www.sciencedirect.com/science/article/pii/S1473309920308434" rel="nofollow">https://www.sciencedirect.com/science/article/pii/S147330992...</a><p>[9] <a href="https://journals.sagepub.com/doi/full/10.1177/00375497231152458" rel="nofollow">https://journals.sagepub.com/doi/full/10.1177/00375497231152...</a>