I agree with many of the points in the article, however it is conflating two unrelated things. Using simple averages in isolation is almost always a mistake. The real story in a system almost always lies in it's variation and not in the average. There are many ways to look at that, one of them is a histogram, although I tend to prefer time series graphs with moving ranges. One of my favorites is the Xmr chart or any of a number of statistical process control charts. Using those you can get a much better understanding of systems over time.<p>The transition to Real User metrics is not at all related to using an average. All recording and instrumentation processes will benefit through using more sophisticated and nuanced tools than a simple average. Real user metrics are an important data point, but they lack certain information you can get from external monitoring systems. We use both pingdom and catchpoint, by far my favorite is Catchpoint because I can see things like what ISP is involved with a slow request, what geographic region, etc. I can also get scatter plots and nice statistical graphs around median, geometric mean, 75th, 95th, 99th percentile.<p>So in short the main points are good, simple averages are misleading. Capturing end user performance data is good. Not using external monitoring though isn't a good idea because there a number of things that you can identify if you have that insight.
Disclaimer: My company also does web performance analytics.<p>I did a talk a couple of years ago on the statistics of web performance where I cover things like median, arithmetic mean, geometric mean, margin of error and sample sizes to carry out proper data analysis. Slides available here: <a href="http://www.slideshare.net/bluesmoon/index-3441823" rel="nofollow">http://www.slideshare.net/bluesmoon/index-3441823</a><p>To zashapiro's point about Geometric Mean... while it tends to be superior in the ideal case where the distribution is perfectly Log-normal, on average, most distributions tend to sway from a perfect log-normal. The median gives you a slightly better measure of central tendency in this case.<p>Secondly, there's the problem with user perception of geometric standard deviation (and consequently margin of error). Unlike arithmetic standard deviation which is +-, the geometric standard deviation is */, which means it's not visually symmetric... humans have an easier time visualizing additive symmetry than they do with multiplicative symmetry.<p>At LogNormal.com, we track the median, arithmetic mean, geometric mean a whole bunch of percentiles and margins of error and a complete distribution curve.
Or, put more simply.<p>The average of [0,100,200] and [99,100,101] are both 100. And yet these two data sets are clearly different.<p>Measures of central tendency should always be supported by measures of dispersion (range, standard deviation, etc.) Not just with web analytics.