Why squared error? (2014)

Why squared error? Because you can solve the equation to minimize squared error using linear algebra in closed form.<p>Why L2 regularization? Same reason. A closed form solution exists from linear algebra.<p>But at the end of the day, you are most interested in the expectation value of the coefficient and minimizing the squared error gives you E[coeffs] which is the mean of the coefficients.

There is a Kaggle competition right now that uses mean absolute error, and this makes the problem substantially harder. For a practical discussion of techniques used to solve machine learning problems that use mae see the forums in: <a href="https:&#x2F;&#x2F;www.kaggle.com&#x2F;c&#x2F;allstate-claims-severity&#x2F;forums" rel="nofollow">https:&#x2F;&#x2F;www.kaggle.com&#x2F;c&#x2F;allstate-claims-severity&#x2F;forums</a><p>As touched upon in the article, the objective not being differentiable is a big deal for modern machine learning methods.

For minimizing the square of the errors I think the good reason is because, assuming your data has gaussian probability distribution, minimizing the square error corresponds to maximizing the likelihood of the measurement, as you and others have said.<p>Why do we assume gaussian errors? There is seldom a gaussian distribution in the real world usually because the probability for large error values doesn&#x27;t not decay that fast. We use it because the math is easy and we can actually solve the problem assuming that.

The best explanation is probably that squared error gives you the best fit when you assume your errors should normally distributed.<p>Things like the fact that squared error is differentiable are actually irrelevant - if the best model is not differentiable, you should still use it.

I asked that early in my career.<p>We want a metric essentially because if we converge or have a good approximation in the metric then we are close in some important respects.<p>Squared error, then, gives one such metric.<p>But for some given data, usually there are several metrics we might use, e.g., absolute error (L^1), worst case error (L^infinity), L^p for positive integer p, etc.<p>From 50,000 feet up, the reason for using squared error is that get to have the Pythagorean theorem, and, more generally, get to work in a Hilbert space, a relatively nice place to be, e.g., we also get to work with angles from inner products, correlations, and covariances -- we get cosines and a version of the law of cosines. E.g., we get to do orthogonal projections which give us minimum squared error.<p>With Hilbert space, commonly we can write the total error as a sum of contributions from orthogonal components, that is, decompose the error into contributions from those components -- nice.<p>The Hilbert space we get from squared error gives us the nicest version of Fourier theory, that is, orthogonal representation and decomposition, best squared error approximation.<p>We also like Fourier theory with squared error because of how it gives us the Heisenberg uncertainty principle.<p>Under meager assumptions, for real valued random variables X and Y, E[Y|X], a function of X, is the best squared error approximation of Y by a function of X.<p>Squared error gives us variance, and in statistics sample mean and variance are <i>sufficient statistics</i> for the Gaussian; that is, for statistics, for Gaussian data, can take the sample mean and sample variance, throw away the rest of the data, and do just as well.<p>For more, convergence in squared error can imply convergence almost surely at least for a subsequence.<p>Then there is the Hilbert space result, every nonempty, closed, convex subset has a unique element of minimum norm (from squared error) -- nice.

I am no math expert, but I have always thought about it like this. The squared error is like weighting the error by the error. This causes one big error to be more significant than many small errors, which is usually what you want. Am I on the right track?

Squared error represents the underlying belief that errors in various dimensions, or errors in independent samples, are linearly independent. So they add together like orthogonal vectors, forming a vector whose length is the square root of the sum of the squares. Minimizing the square error is a way of minimizing that square root without the superfluous operation of calculating it.

It&#x27;s fine to list some reasons for using squared error, but you really can&#x27;t decide on the error function without referring to a problem you&#x27;re trying to solve.<p>Just look at the success of compressed sensing, based on taking the absolute value error seriously.

&quot;inner products&#x2F;gaussians&quot; - the absolute value (and also cuberoot of absolute cubes, fourth root of fourth powers) also define inner products. Likewise, there are &quot;gaussian-like formulas&quot; which take these powers instead of squared.<p>However: if you look at the shape of the squareroot of sum squares, it&#x27;s a circle, so you can rotate it. If you take the absolute, it&#x27;s a square, so that cannot be rotated; the cuberoot of cubes and fourthroot of fourths, etc. look like rounded edge squares, and that cannot be rotated either, so if you have a change of vector basis, you&#x27;re out of luck.<p>With the gaussian forms of other powers, none of them have the central limit property.

The Bayesian formulation for the likelihood function would make this squared error explicitly clear.

My explanation for squared error in linear approximation always was: because it minimizes the thickness of the line that passes through all the data points.<p>(Per the old math joke - you can make a line passing through any three points on a plane if you make it thick enough.)

Or why use variances when there are standard deviations (the square root of the variance) which have more easily interpreted units? One commonly cited reason is that one can sum variances from different factors, which one cannot do with standard deviations. There are other properties of variances which make them more suitable for continued calculations[1]. This is why, for instance, variances are often utilized in automated optimization packages.<p>[1] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Variance#Properties" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Variance#Properties</a>

Interesting discussion. Not sure about the breakdown between ridge regression and LASSO though. The difference is not in the error term but in the regularization term.

Squared error because the uncertainties in independent, normally distributed random variables add in quadrature. I expect that this could be proved geometrically using Pythagoras&#x27;s theorem, so in that sense the comments about orthogonal axes are vaguely on the right track.<p>Normally distributed variables because the central limit theorem.<p>It isn&#x27;t all that complicated.

Why not KL-Divergence, which measures the error between a target distribution and the current distribution? From the perspective of Information Theory, it is the best error measurement.<p>Oh, and let&#x27;s not forget that for a lot of problems minimizing the KL-divergence is the exact same operation as maximizing the likelihood function.

Another pro tip - absolute error magnitude is the convex hull of non-zero entry count for vectors (l_0 norm in some circles). So in the convex minimization context (and for most other smooth loss terms in general) you end up with solutions with more zero entries and few possibly large non-zero entries.

Also see <a href="http:&#x2F;&#x2F;www.leeds.ac.uk&#x2F;educol&#x2F;documents&#x2F;00003759.htm" rel="nofollow">http:&#x2F;&#x2F;www.leeds.ac.uk&#x2F;educol&#x2F;documents&#x2F;00003759.htm</a>.

Somewhat related; here&#x27;s my attempt at explaining Cross Entropy:<p><a href="http:&#x2F;&#x2F;heliosphan.org&#x2F;cross-entropy.html" rel="nofollow">http:&#x2F;&#x2F;heliosphan.org&#x2F;cross-entropy.html</a>

Square often corresponds to power in systems.

Why geometric mean?, I would ask.

cause linear algebra is a beautiful framework to think in.

To make it positive and to amplify it (as a side-effect).<p>BTW, &quot;error&quot; is a misleading term - it communicates some fault, at least in the common sense. Distance would be much better term.<p>So, &quot;squared distance&quot; makes much more sense, because negative distance is nonsense.

An honest question - do we even need statistics when we have machine learning? Statistics to me appears as a hack&#x2F;aggregation of data we couldn&#x27;t process at once in the past; these days ML + Big Data can achieve that and instead of statistics we can do computational inference instead. To me this looks like looking back to &quot;old ways&quot; for a reference point instead of looking forward to the unknown but more exciting.