Always-On Time-Series Database: Keeping Up Where There's No Way to Catch Up

ACM Queue interview of Theo Schlossnagle, founder and CTO of Circonus, about IRONdb [1], an alternative time-series platform to Prometheus&#x2F;InfluxDB. Some high-level architecture choices:<p><pre><code> - CRDT to avoid coordination service - LMDB (B-Tree) for read optimized requests - RocksDB (LSM-tree) for write optimized requests - Modified consistent hashing ring split across two Availability Zones - OpenZFS on Linux - Flatbuffers for Serialization-Deserialization - Information Lifecycle Management for historical data - circllhist; HDR [high dynamic range] log-linear quantized histograms </code></pre> [1] <a href="https:&#x2F;&#x2F;docs.circonus.com&#x2F;irondb&#x2F;" rel="nofollow">https:&#x2F;&#x2F;docs.circonus.com&#x2F;irondb&#x2F;</a>

&gt; But the real advantage of the CRDT approach is that, if you can limit your entire operation set to CRDTs, you can forego consensus algorithms such as Paxos, Multi-Paxos, Fast Paxos, Raft, Extended Virtual Synchrony, and anything else along those lines.<p>Things like this are said a lot, but I don&#x27;t believe CRDTs provide the same consistency guarantees as Paxos&#x2F;Raft.<p>Anyone have thoughts on why CRDTs might be favored over the internals of something like a well written eventually consistent datastore like Cassandra that doesn&#x27;t use Paxos&#x2F;Raft?<p>Is he just saying that your consensus algorithm then just becomes that math of CRDTs and not the careful complexity of a distributed consensus protocol?<p>(the interviewee then goes on to describe that CRDTs fit their use case well, so maybe that is all he is saying...)

&gt; We&#x27;ve witnessed more than exponential growth in the volume and breadth of that data. In fact, by our estimate, we&#x27;ve seen an increase by a <i>factor of about 1x10^12</i> over the past decade.<p>If 10 years ago we had X data collected, now it is 1,000,000,000,000 * X data??? I find this highly surprising and wonder if it is a typo, and if so, what the real numbers are.