Beyond NoSQL: Using RRD to store temporal data

Non sequitur hint: If you are storing data like his powerhungy, consider storing the sum of the squares of the datum as well as the datum and number of samples in the aggregate (might be initially 1). This lets you compute the standard deviation for display, but it also has the nice property that after aggregating samples, you can still compute the standard deviation of that.

RRDtool is pretty nice, but it has a fair number of scalability issues too:<p>* Once you create an RRA (archive file) you can't modify it to add or remove metrics, or change their properties. This makes them relatively inflexible.<p>* Updating RRAs is I/O heavy. Every time an update comes in, the OS must read, modify and write a page.<p>* RRDcache mitigates this somewhat by deferring flushes, but there are diminishing returns to this (eventually the number of writes coming in will cause the cache flush and filesystem metadata update rate to exceed the maximum IOPS available), and you risk data loss in the event of a power outage or the OOM killer kills the process.<p>Time-series data access patterns tend to be write-heavy. Storing first in an append-only log is a big win here; Cassandra and MySQL are both good choices, though you do have to think about the schemata first. And disk is so cheap now that expiration can be an afterthought.

Site appears to be temporally unavailable.

You could do the same thing with mongodb and 'capped collections' although aging the data like rrd would require mongodb to have a callback for when the capped collection is full.

That was one of the clearest explanations of the strengths of RRDtool that I've read. You can spend a lot of time massaging a more general database to store time series data, or you can use RRDtool.

Pity there's no mention that RRDTool has been around for decades, pretty much stable. It's worth remembering that old tools aren't necessarily obsolete.