The article starts with, "Human beings try to find patterns to explain the reason behind almost every phenomenon, but that doesn’t mean that there is a pattern to rely on." The second part of the sentence is true, on further observation some patterns prove not to be patterns. However the first half, the attempt to explain <i>why</i> we "find patterns" isn't convincing.<p>Instead, replace that phrase with this: "based on brain construction, we humans are predisposed to find patterns in data we encounter". This idea holds up to evolutionary scrutiny, organisms have mechanisms biased for self-protection and finding food and other resources through various forms of pattern recognition. IOW we find patterns we're neurologically capable of finding, particularly in regard to survival and reproduction.<p>Perhaps it's more accurate to assert we "find patterns" useful in making predictions about our present and future state within the environments we occupy. The fact that patterns may not turn out to be authentic is simply part of a process of refinement of pattern-seeking and improving value as predictors of future states. We may call a pattern an "explanation" but nothing is actually "explained" as shown by the fact we reserve the right to enhance or revise the "pattern", or what we insist it predicts, at any given time.<p>Edits: grammar and clarity
>As an analyst, one needs to keep in mind that the Journey is more important than reaching the Destination.<p>I'm really not sure what to make of the last line. The goal of analysis should be to produce results that are actionable. In the end it should matter very little how they are obtained as long as they are accurate.
I wish the article was on Apophenia, it would be more fun.<p>Instead it's not clear to me who the target audience is. The phrasing makes it appear to be targeted at analysts and not their business partners. Assuming that to be the case, senior analysts are substantially beyond the level this article is written at (or should be).<p>Entry level analysts need close supervision to prevent them from making these, and other, mistakes. The examples the author draws (specifically cheese vs. infant mortality and the google flu approximations) don't do a good job at identifying when this issue arises. For the cheese example it's unclear if the phenomenon is real or not (the magnitude of the variation in infant death may actually be significant, if the cheese consumption variation was small then there would be a different story). The author does nothing to help the reader resolve this.<p>In the Google flu example it's only through hindsight (and colossal failure) that the author identifies the lack of validity in Google's model.<p>I agree 100% with his point but I don't think the article is providing much value because essentially the author is simply saying: "be aware, this type of problem exists out there..." without providing information necessary to navigate/resolve the problem.
Can anybody reading this comment please enlighten me on good algorithms for optimum bin sizes for histograms? I have tried DW Scott's (1979) method. But are there any new better kid in the block?
Our brains are wired to believe a plausible story easier than the one told by the base rate or stats... We need to be aware of the gaps our brain fills by itself. Definitely not easy, though...