We're increasingly aware today of how the media operates cycles of self-referential and self-justifying citations: a TV show will quote an article that reports "some people" taking an issue, which ends up being a quote from someone interviewed for another newspaper article.. and so on. This "legitimacy laundering" is rampant, and we're now getting towards media literacy levels which expose it for many people.<p>However, most concerning: this is how academia has always worked. It's the great absurdity of <i>peer</i> review, and of citation. This is how entire fields can sustain themselves with little or no scientific validity (esp. see, psychometrics).<p>We are no where near the equivalent "academic literacy" for generally informed members of the public to understand this problem. <i>Entire</i> fields can be sustained with zero "empirical pressure" to close down. So long as one can cite another who can cite another... and somewhere some government body will take these citations as prima facie evidence of a research programme, then funding will be given and more papers published.
Besides the sociological problems listed, we must always be conscious of how counterintuitive and difficult statistical inference itself can be. Good things to search for are statistical fallacies, probabilistic paradoxes and books like Counterexamples in Probability.<p>And it is not sufficient to read about them once or twice; researchers who use statistical inference regularly must revisit these caveats at least as regularly.<p>Myself, I have taught Probability and Statistics many times, discussed and dispelled many misconceptions by students. Would I be 100% sure I will not be caught up in a fallacy while informally thinking about probability? I wouldn't even be 10% sure; any intuition I conjure up, I would triple check as rigorously as possible.
On page 11 there is a mention of taking the result of self reporting (surveys) at their word. I’ve wondered about this issue not just in science but other situations. For example political polling, data point in time surveys, census, etc. Without verification, what good is the data? And yet you often see such self reported data quoted by articles or papers as if it were factual.