Why would you choose a LSM Tree based storage mechanism for a message queue?<p>The only reason I can come up with would be because it's a read-to-use library you can just plug in which gives OK performance and some handy features because you can use the KV store for other things. But it doesn't scale well and backups with LevelDB are not really easy either (close DB, copy all files).<p>Message queues when they are ordered (at least on the local node/queue level) usually just need some kind of append-only log file. You don't do random reads or writes into the middle of the queue, you only modify the head and tail.<p>InfluxDB, albeit being a time series db has similar write patterns to a message queue, learned it the hard way when they first tried to use a LSM Tree database (LevelDB), then switched to a B+Tree (BoltDB/LMDB) but that also doesn't scale once the DB gets big and the tree has quite some depth.
They kindly did a nice writeup of their journey: <a href="https://influxdb.com/docs/v0.9/concepts/storage_engine.html" rel="nofollow">https://influxdb.com/docs/v0.9/concepts/storage_engine.html</a><p>Why not do it simple and use append-only files without complex structure and management?<p>Check out Kafka for a better storage format for message queues of this kind.<p>PS: every message queue should first clearly explain what guarantees it provides.
This couldn't have come at a better time - I was actually looking for a durable message-queue written in Go. Is there any way to read more about the architecture of this system? I find systems like these to be quite fascinating but taking the time to go through the code can sometimes be very time-consuming. It would be awesome if more projects have a writeup as detailed as cockroachdb[0]!<p>Aside: There used to be a site sometime back which used to distribute compiled binaries of Go code for all platforms? Is it still up any chance?<p>[0] - <a href="https://github.com/cockroachdb/cockroach#architecture" rel="nofollow">https://github.com/cockroachdb/cockroach#architecture</a>
Sounds interesting. For my usecases, which require few (< 10) messages/sec and no clustering, would I gain anything by using Siberite over Beanstalk?