The Rise of Open Source Time Series Databases

If you are writing an article about time series databases, and you don&#x27;t mention KDB - straight to jail. It is the grandfather of time series and predates influx by about a decade. It is still the fastest out there too. It is used by about every major financial and trading institution in the US and Europe.<p>Everybody thinks TSDB are something new-ish, but they&#x27;ve been around since the days of APL. All you youngins disappointment me every time you write about time series, vector languages, or data-oriented programming and entirely neglect all the work that comes under the APL&#x2F;Vector umbrella. SOA and DOD have been around for 50+ years, and they didn&#x27;t start with c++ or Pandas.<p>Now the creator, Arthur Whitney, has a new one out called Shakti that is even faster (but has also ditched from the &quot;niceties&quot; of Q.<p><a href="https:&#x2F;&#x2F;shakti.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;shakti.com&#x2F;</a>

If the author ends up reading this... as someone who is ignorant but curious about time series databases, it was frustrating to see an example of what relational data looks like, but then no example of what time series data looks like.<p>It might take a bit more context or setup, but this article is an opportunity to educate folks who aren&#x27;t deep in the same trench.<p>Edit: in fairness, this article doesn&#x27;t describe itself as &quot;a Postgres developer&#x27;s guide to getting started with time series databases&quot; and maybe that&#x27;s what I was unfairly hoping I would find.

Had me until claiming that InfluxDB was the first mainstream TSDB <i>in 2013</i>. OpenTSDB (2010)? Graphite (2008)? RRDtool (1999)?<p>Maybe Influx took off in a way these prior projects didn&#x27;t, but people have been storing time series data for decades.

This is unsurprisingly a VictoriaMetrics post that frames the history in a narrow way to talk favorably about VictoriaMetrics.

A bit opinionated and partial version of history (graphite was big before Influxdb and Prometheus came out, and the landscape is bigger than what they mentioned) but it might be a good enough starting point to learn about this.

The author calls out InfluxDB as the first time series database.<p>Why would that not have been rrdtool or something earlier?

They are definitely using a more narrow definition of time series database than I expected - they don&#x27;t mention rrd for example, which is GPL and been around for decades.

Wow this article misses many TSDBs that predated those listed. Victoria seems to be basically a rewrite of Prom.

Graphite[1] has interesting plugin ecosystem like Grafana[2] &amp; has been around since 2009. Architecture of Open Source Applications book[3] has done a nice overview of the architecture. I really like how it reuses linux filesystem as data store, making archiving very simple.<p>[1]: <a href="https:&#x2F;&#x2F;graphiteapp.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;graphiteapp.org&#x2F;</a> [2]: <a href="https:&#x2F;&#x2F;grafana.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;grafana.com&#x2F;</a> [3]: <a href="https:&#x2F;&#x2F;aosabook.org&#x2F;en&#x2F;v1&#x2F;graphite.html" rel="nofollow">https:&#x2F;&#x2F;aosabook.org&#x2F;en&#x2F;v1&#x2F;graphite.html</a>

I&#x27;m using VictoriaMetrics (VM) to store basic weather data, like temperature and humidity. My initial setup was based on Prometheus however it seemed very hard to set a high data retention value, default was something like 15 days if I recall correctly.<p>Since I would actually like to store all recorded values permanently, I could partially achieve this with VM which let me set a higher threshold, like 100 years. Still not &#x27;forever&#x27; as I would have liked, but I guess me and my flimsy weather data setup will have other things than the retention threshold to worry about in 100 years.<p>Would be nice to learn the reason why an infinite threshold is not allowed.

My biggest beef with VictoriaMetrics is I can&#x27;t delete and re-ingest a range of data. i.e. if I discover yesterdays data was messed up due to some issue in downstream data processing, I can&#x27;t delete yesterdays data and re-ingest the cleaned up data. (Or even just delete yesterdays bad data)<p>The only workaround I&#x27;ve found so far is to dump out the whole timeseries which could go back months&#x2F;years, delete the timeseries, fix&#x2F;delete data in the dump and re-ingest the whole thing. This really, really sucks.

How do these stack up to <a href="https:&#x2F;&#x2F;www.timescale.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.timescale.com&#x2F;</a> ?

Why would you use this instead of CH if your usecase is metrics?

Interesting! Too bad it’s just about two of them.

InfluxDB is open source (<a href="https:&#x2F;&#x2F;github.com&#x2F;influxdata">https:&#x2F;&#x2F;github.com&#x2F;influxdata</a>) and seems to be leader by far as per DBEngines ranking : <a href="https:&#x2F;&#x2F;db-engines.com&#x2F;en&#x2F;ranking&#x2F;time+series+dbms" rel="nofollow">https:&#x2F;&#x2F;db-engines.com&#x2F;en&#x2F;ranking&#x2F;time+series+dbms</a>

I might expect that for a specialized task, specialized database technology would outperform general relational databases. But what specific technologies have significant impacts on time series db efficiency?

I wonder if the in-house metrics systems at big tech firms like Google and Facebook should be counted as ‘proprietary’ for these purposes. I suppose not because one can’t really pay to get them internally.

Can anyone comment why these are superior to ClickHouse? I really like the ClickBench which compares the various products performance (and open source).

This topic has been done to death. Too many OSS options in the last ~10 years with very little differentiation.<p>Let&#x27;s talk LLMs instead.