TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

The Easiest Data Analysis Mistake to Make

44 pointsby lejohnqover 11 years ago

8 comments

lpageover 11 years ago
This is a very simple and very pathological example that&#x27;s easily ferreted out with a few more summary statistics (median, min, max) but it&#x27;s a good illustration of the blind application of statistics. Short of visualization, non-parametric statistics really help with such things. Correlation is a fragile, linear measure, and things that are obviously correlated by inspection can easily appear mathematically uncorrelated -- points on a unit circle, for example. Likewise, the mean of any skewed distribution tells you very little, but that&#x27;s the statistic that&#x27;s always cited. Quantiles, medians, and non-parametric measures of correlation such as rank correlation are simple and often overlooked. They do a good job screening for pathological data sets like Anscombe&#x27;s quartet and real world ones.<p>It&#x27;s also worth mentioning &quot;dumbbell&quot; data sets. Two clusters of data, each of which have a independent, meaningful correlation in them, can easily leverage a linear regression into a meaningless line passing through the two clusters. That&#x27;s a pretty common issue with high dimensional data (obviously you can see it in a 2D scatter plot), and it&#x27;s not easily caught short of looking at regression diagnostic statistics.
评论 #6891898 未加载
brian_peirisover 11 years ago
I think, typically, if you&#x27;ve gone to the trouble of calculating variance and correlation, you would have also calculated the median and mode of these datasets. The differences would have been obvious with those basic analyses.
评论 #6893266 未加载
carlosggover 11 years ago
<a href="http://en.wikipedia.org/wiki/Anscombe&#x27;s_quartet" rel="nofollow">http:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Anscombe&#x27;s_quartet</a>
mrcactu5over 11 years ago
Statwing looks great!<p>Right now I do my data analysis in numpy, but this looks good for my Excel-based colleagues.<p>What library is doing the statistics?
评论 #6891749 未加载
timrufflesover 11 years ago
So a concept you find in a beginner stats textbook is now &#x27;news&#x27;? Definition of blogspam surely...
评论 #6891987 未加载
davidmanescuover 11 years ago
My favourite part of this (aside from the message) is that it links back to this exact discussion page.
ejainover 11 years ago
An equally common mistake is to visualize without analyzing :-)
medaganover 11 years ago
who doesn&#x27;t look at the data range, min, max, mode???
评论 #6891815 未加载
评论 #6891611 未加载