Turning the database inside-out with Apache Samza

Immutability is hardly a cure-all, see the discussion here for why RethinkDB moved away from it: <a href="http://www.xaprb.com/blog/2013/12/28/immutability-mvcc-and-garbage-collection/" rel="nofollow">http:&#x2F;&#x2F;www.xaprb.com&#x2F;blog&#x2F;2013&#x2F;12&#x2F;28&#x2F;immutability-mvcc-and-g...</a><p>The reality is shared, mutable state is the most efficient way of working with memory-sized data. People can rant and rave all they want about the benefits of immutability vs mutability, but at the end of the day, if performance is important to you, you&#x27;d be best to ignore them.<p>Actually, to be more honest, reality is more complicated still. MVCC that many databases use to get ACID semantics over a shared mutable dataset is really a combination of mutable and immutable.

<i>... most self-respecting developers have got rid of mutable global variables in their code long ago.</i><p>I&#x27;m not convinced that&#x27;s the case. Almost everyone has merely hidden their mutable globals under layers of abstractions. Things like &quot;singletons&quot;, &quot;factories&quot;, &quot;controllers&quot;, &quot;service objects&quot;, &quot;dependency injection&quot; are the vernacular of the masked-globals game.

As one who works with analytics databases and ETL (extract-transform-load) processes a great deal, immutability of data stores is an incredibly valuable property. Maybe append-only does not make sense in operational databases all the time, but for non-real-time analytics, it makes a huge amount of sense. In my case, operational data is queried, optimized for storage space and quick loading, and cached to disk. Because it is an analytics database used for longer-term analysis and planning, daily queries of operational data are sufficient in many cases. Operational workload is not even a consideration. The ETL process also allows for &quot;updating&quot; records in the &quot;T&quot; (transform) part. Updates to operational data are not even necessary, and often impossible, so correcting and enhancing the data for decision making is a huge win for clients. Issues similar to &quot;compaction time&quot; can still occur, but an ETL approach allows for many clean ways of controlling the process and avoiding those failure scenarios.

Anyhow in the Bay Area interested in learning more about Apache Samza should attend the meetup tonight in Mountain View: <a href="http://www.meetup.com/Bay-Area-Samza-Meetup/events/220354853/" rel="nofollow">http:&#x2F;&#x2F;www.meetup.com&#x2F;Bay-Area-Samza-Meetup&#x2F;events&#x2F;220354853...</a>

I&#x27;m not sold on Samza, but I can tell you that creating isolated services that create their datastore from a stream of events is a really useful pattern in some use cases (ad-tech).<p>I&#x27;ve made use of NSQ to stream user update events (products viewed, orders placed) to servers sitting at the network edge which cache the info in leveldb. Our request latency was something like 10 microseconds over go&#x27;s json&#x2F;rpc. We weren&#x27;t even able to come close to that in the other nosql database servers we tried, even with aggressive caching turned on.

Similar interesting talk by Rich Hickey:<p><a href="http://www.infoq.com/presentations/Value-Values" rel="nofollow">http:&#x2F;&#x2F;www.infoq.com&#x2F;presentations&#x2F;Value-Values</a>

You can do similar &quot;magic&quot; cache invalidation with Elasticsearch and the percolate feature. Each time you do a query and cache some transformation of the result, put that query in a percolate index. Then when you change a document, run the document against the percolate index and, voila, you get the queries that would have returned it and can then invalidate your cache.<p>This method of cache invalidation fails in a very key place though (just like in the article). What happens if you change a very core thing that invalidates a large percentage of the cache?

This is a cool idea - the holy grail scenario I&#x27;m envisioning is storing all data in the log i.e<p>1. the transaction log is a central repository for all data 2. much more detailed data is stored, enough that analytics and can run off this same source of data<p>The amount of data generated increases proportional to the number of updates on a row&#x2F;piece of data whereas with a mutable solution, it is constant w.r.t number of updates on the same data. That is a pretty big scaling difference.<p>However, storing that much data translates to much higher costs for HDDs&#x2F;servers, or possibly lower write performance if the log is stored on something like HDFS.<p>There would also be performance costs for building and updating a materialized view. Imagine a scenario like this:<p>Events -&gt; A B C D E F G H I J K Materialized view M has been computed up to item J (but not K yet) Read&#x2F;Query M<p>Now either writing K incurs the cost of waiting for all dependent views to materialize, or the read on M incurs the cost of updating M.<p>Some fusion of this would be pretty interesting though. For example, what if we just query on M without applying any updates if there have been &lt;X updates? That translates to similar guarantees as an eventually consistent DB - the data could be stale. Atleast it gives us more control over this tradeoff.

I really enjoyed reading about Storm too: <a href="http://nathanmarz.com/blog/history-of-apache-storm-and-lessons-learned.html" rel="nofollow">http:&#x2F;&#x2F;nathanmarz.com&#x2F;blog&#x2F;history-of-apache-storm-and-lesso...</a><p>This kind of &quot;competition&quot; leads to analysis paralysis though. Its much better when there is a single winner...

<i>A more promising model, used in some systems, is to think of a database as an always-growing collection of immutable facts.</i><p>That would already be a huge progress over how databases are currently used; if records were in fact immutable many problems would be instantly solved.

Does anyone knows which app has been used to create the &quot;handwritten&quot; images? I draw very badly so I&#x27;m looking for such an app to explain data flows on a corporate blog&#x2F;wiki.

Streams - another reinvention of LDAP Persistent Search.<p>Yes, there really <i>are</i> protocols that handle single request&#x2F;multiple response interactions, and they&#x27;ve been around for decades. Unlike crap built on HTTP, which was never intended for uses like this, these protocols work well with multiple concurrent requests in flight simultaneously, etc.

Conceptually, one of the challenges of streams as first class citizens is that humans don&#x27;t do well with them. For the purposes of analysis, humans need a &quot;snapshot&quot; or fix on the data. This way they can derive insights from the data and act on human things. The reality is that, for many real-world scenarios, a real-time view of the data is not just a luxury, it&#x27;s actually a drawback, because data changes are noisy. Many human problems deal with abstract representations of the actual data, and so imprecision is part of the problem.<p>I really like the talk from the point of view of simplifying the system-wide problems caused by a gigantic mutable state. But I feel that at the border of system to humans there will be other issues to discuss.

This is CouchDB, right?