Scaling Analytics at Amplitude

What shortcomings of Redis set operations does the in-memory data store address, and how?<p>Unrelated rant: regardless of its merits, &quot;Lambda&quot; Architecture is probably the most annoying overloaded term in use today, second only to &quot;Isomorphic&quot; Javascript. Just because something has a passing resemblance to the functional style doesn&#x27;t grant license to re-appropriate a well understood term of art.

Out of curiosity why weren&#x27;t products like Druid <a href="http:&#x2F;&#x2F;druid.io&#x2F;" rel="nofollow">http:&#x2F;&#x2F;druid.io&#x2F;</a> or influxdb <a href="https:&#x2F;&#x2F;influxdb.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;influxdb.com&#x2F;</a> or possibly opentsdb taken into consideration ?

&quot;Finally, at query time, we bring together the real-time views from the set database and the batch views from S3 to compute the result&quot;<p>so how in the heck does this work? at query time you decide what file to get our of s3 (hwo do u decide this?), parse it, filter it, and merge with the results from the custom made Redis like real time database?

Author of the post here. Happy to talk about how we&#x27;ve designed&#x2F;built our architecture at Amplitude!

&gt; in-memory database holds only a limited set of data<p>MemSQL is not just in-memory, but also has column-store (note: I don&#x27;t know VoltDB). You can think of MemSQL not as &quot;does everything in-memory&quot;, but &quot;uses memory at the best&quot;.

How do you decide what sets of users you pre aggregate?<p>It seems like without some limits in place you could end up with huge number of sets, especially if you are calculating these based on event properties.

Do you store raw data ingested from Kafka directly in S3 or have an intermediate database for hot data?

Its easy to see how you can calculate segments, retention and trends using user sets, but how do you calculate funnels.

The lambda architecture and the split between the heavy slow processing and the interactive processing reminds me of how a few of our customers are blending Hadoop and Couchbase for similar use cases: <a href="http:&#x2F;&#x2F;www.couchbase.com&#x2F;fr&#x2F;ad_platforms" rel="nofollow">http:&#x2F;&#x2F;www.couchbase.com&#x2F;fr&#x2F;ad_platforms</a>