Introducing Streaming K-Means in Spark MLlib 1.2

This is a cool feature, and is one of the prime example of what Spark's tight integration of various libraries can enable (in this case Spark Streaming and MLlib). It was originally designed by Jeremy Freeman to handle workloads in neuroscience, which IIRC was generating data at 1TB/30mins.

Sounds similar to an exponential moving average[1], which itself is a one-pole IIR digital filter. [1] <a href="http://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average" rel="nofollow">http:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Moving_average#Exponential_movi...</a>

Is it true that this doesn&#x27;t support dynamic values of k? That is, the algorithm isn&#x27;t adaptive to a changing number of clusters? That said, I suppose for some small range of k values, you could do this trivially by tracking them all and picking the best.

very interesting post. ironically, hackernews uses a similar type of time-decay algorithm!