A related really cool thing I learned in grad school is that you can implement Deming regression (<a href="https://en.wikipedia.org/wiki/Deming_regression" rel="nofollow">https://en.wikipedia.org/wiki/Deming_regression</a>) by storing the moments (outer product sum and point sum) of the training points and then finding the dominant eigenvector of the outer product sum using a singular value decomposition. Since you can approximate the directional part of a 2x2 SVD with atan2(), it effectively becomes a O(1) operation to add or remove training points.<p>(specifics here: <a href="https://april.eecs.umich.edu/courses/eecs568_f12/linefitting.pdf" rel="nofollow">https://april.eecs.umich.edu/courses/eecs568_f12/linefitting...</a> )