Despite Segment using Kafka nowadays, I would encourage everyone to give NSQ a try and see if it might not already fit your use-case. For me it's been one of those do-one-thing-well pieces of software that's really well-engineered, where I can clearly understand how it works and what the scope of its features are.<p>NSQ is my go-to queue whenever I'm working on any kind of service, even more so when that service might be standalone. Consider deploying a service on an isolated box which frequently pushes data into the cloud. Connectivity to the cloud might be intermittent, so you just store every work item to the locally running nsq and you're protected against network outages as well as your data-producing process crashing. Since NSQ is so easy to spin up and easy to interact with (Publish works via plain HTTP), it's not a lot of overhead to add it to your process.
I'm working at Segment now as a Software Engineer in Infrastructure. We just went through a big effort to remove NSQ (in favor of Kafka). Happy to answer any questions.
Been using nats for a few projects lately and really enjoying it. One of the many great golang projects. Resource utilization compared to kafka is tiny.<p>Re: msq: Looks like segment are using kafka these days despite the praise for nsq in this dated article.