After trying InfluxDB and it not working well for my use case I've written a TSDB as well. I suspect that's the joke around these parts, everyone and their cousin has written as TSDB.<p>Used time based blocks just like the article says. But they also switched if they got larger than 2GB, because it allowed the index offset to be 4 bytes. That allowed pretty fast searching as well as discarding old data.<p>The data and index and were both written to separate files. Some of the values were not just simple integers/floats but large blobs that's why separate data files.<p>Decided not to go explicitly with mmap-ing on read. Started that way but abandoned it. A simple pread (read + seek in one syscall) and relying on page caching worked just as well.<p>Should have used an inverted index for labels, that's a good idea. Maybe just sqlite or something like that... Though my queries were not as free form and some were more common than others. So built custom indices for the common queries.<p>Index files were also partitioned by time but were synchronized to the same partitioning schedule as the main data file. So if main data file opened a new block file, indices did the same.<p>Didn't explicitly handle in memory vs not-in memory blocks when writing, but relied on periodic fsync-ing.<p>Instead of a wal decided that some data that wasn't fsync-ed yet might be lost if power was cut. To deal with corruption on start, data in main file and extra indices were truncated to lasted sane fsync of the main time based index.
When I read this blog post, it convinced me to use prometheus / grafana. I'm now monitoring over 1000 Pi class boxes using a small custom golang agent running on each one. Things have been rock solid. Couldn't be happier with this solution. Thanks for all the hard effort.