I am looking for a program that is kind of like a message broker like RabbitMQ but without the fancy routing capabilities and just a message queue.<p>I want to stream several types of messages from one data center to another with very high throughput. The system needs to support compression, SSL encryption, and needs to be able to spill to disk as a cache if the network connection drops or is slow and attempt re-connection. No need for replay support or absolute guaranteed delivery.<p>Are there any plug and play software solutions for this? Obviously I can write these types of features into my existing software but would prefer not to.
NATS is another option but I think your requirements would lead you to NATS Streaming:<p><a href="https://nats.io/documentation/streaming/nats-streaming-intro/" rel="nofollow">https://nats.io/documentation/streaming/nats-streaming-intro...</a><p>It doesn't offer compression as far as I know (the FAQ suggests though that one is free of course to do compression/decompression above NATS).
nsq.io looks interesting. @patio11 used it on Stockfighter (<a href="http://www.kalzumeus.com/2015/08/20/designing-and-building-stockfighter-our-programming-game/" rel="nofollow">http://www.kalzumeus.com/2015/08/20/designing-and-building-s...</a>).