CERN swaps out databases to feed its petabyte-a-day habit

I also dropped InfluxDB at work due to its terrible performance. VictoriaMetrics is great<p>I was using Promscale (TimescaleDB) but they EOL&#x27;d Promscale which forced us to Victoria. But either way both of these are much faster than Influx<p>Don&#x27;t get fooled into the latest InfluxDB rewrite. I think the latest is cloud hosted only too? So stupid

One of my fondest memories as a summer student at CERN in 1993 (in the Electronics and Computing for Physics department) was the visit to the basement beneath the main computing facility, where a colossal tape robot was in operation. Even at that time, CERN was grappling with exceedingly vast amounts of data.

I really like VictoriaMetrics&#x27;s architecture<p>vmagent takes care of all the pesky edge things like emulating prometheus config parsing and various scraping bits. It also does buffering in case you lose network connection for a while, and accept vast spread of different protocols<p>vminsert&#x2F;vmselect scale separately from eachother and your queries don&#x27;t bother your ingest all that much.<p>vmstorage does just that, storage. Only thing that bothers me (compared to say, Elasticsearch), is that data can&#x27;t migrate between nodes so you can&#x27;t &quot;just&quot; start a new one and drain an old one, but a tiny bit ops work in rare cases is IMO price worth paying for straightforwardness of the stack..<p>PromQL compatibility is also great, tools like Grafana &quot;just work&quot; without anyone having to write support for it.<p>We started migrating from InfluxDB at work, and on my private stuff I already did. Soo much less memory usage too.

Missing from the title: leaving InfluxDB and Prometheus for VictoriaMetrics.

That’s one hell of an endorsement. Marketing team won the jackpot.

tl,dr:<p>Speaking to The Register, Roman Khavronenko, co-founder of VictoriaMetrics, said the previous system had experienced problems with high cardinality, which refers to the level of repeated values – and high churn data – where applications can be redeployed multiple times over new instances.<p>Implementing VictoriaMetrics as backend storage for Prometheus, the CMS monitoring team progressed to using the solution as front-end storage to replace InfluxDB and Prometheus, helping remove cardinality issues, the company said in a statement.

We&#x27;ve been using and building with VictoriaMetrics for a while at Batteries Included. I have probably created and torn down 100+ clusters now. It&#x27;s a remarkably easy to use piece of software for something with the capabilities.

At the end of the article it says<p>&quot;<i>InfluxDB said in March this year it had solved the cardinality issue with a new IOx storage engine.</i>&quot;<p>Does this mean that in the end it wasn&#x27;t really necessary to switch to VictoriaMetrics&#x27; offering?

Over 24hr period its more then 11 Gigabytes&#x2F;second or rounding to 100 gbps. Those shards must be pretty crazy

Weird that the title talks about the petabyte a day, while the article is actually about their monitoring tooling, not the thing ingesting the data from experiments, iiuc.

How come they don&#x27;t support wire protocols for analytical workloads like arrow streaming or others like Clickhouse. Looks like they don&#x27;t want to compete with CK.

I’m amazed years after I heard about it how the tiny VictoriaMetrics team is thumping what far bigger organizations manage to do. The biggest reason I’ve heard people don’t want to adopt it is if the lead maintainer gets bussed the project is liable to fall apart.<p>I looked at some of the alternatives to victoriametrics for Prometheus and they all seem… much much worse…

I’m surprised nobody has mentioned grafana mimir yet. You get all the niceties of the prometheus ecosystem with a backend which can scale into the billions of metric streams.

This is nothing compared to what dragnet surveillance has to deal with.

Amazing how much data you can generate with a small cluster piping out &#x2F;dev&#x2F;urandom continuously over every possible socket.

Do not have very positive experience with influxdb, but the strange for me that clickhouse was not even mentioned in the article

Just use sqlite amirite

OPENSOURCE, APACHE2 LICENSE<p><a href="https:&#x2F;&#x2F;github.com&#x2F;VictoriaMetrics&#x2F;VictoriaMetrics&#x2F;blob&#x2F;master&#x2F;LICENSE">https:&#x2F;&#x2F;github.com&#x2F;VictoriaMetrics&#x2F;VictoriaMetrics&#x2F;blob&#x2F;mast...</a>

I can do this on my laptop<p>&#x2F;tumbleweed...