NSQ: Realtime distributed message processing at scale (in Go)

Was something like MQTT investigated prior to re-inventing the wheel with NSQ? From a cursory examination, I don't see any major advantages other than nice UI tools.<p>The wire protocol seems more complicated -- why distinct protocols for producers and consumers? Linebreak-delimited headers are error-prone and ought to be banished. Why transmit the hostname with sub requests -- is another system requesting jobs on behalf of workers?<p>Then the topic/channel distinction seems artificial when wildcards would suffice (and provide much more flexibility), eg topic/*/channels or topic/channels/# in the MQTT parlance. MQTT also has more fine-grained delivery guarantees via its QoS levels. All this with a header structure that's as small as two bytes.<p>edit: While MQTT is pubsub, as long as the system is under your control, you're free to change the semantics at the broker side from "broadcast messages to all consumers" to "rotate messages among consumers."

We're currently using Apache Kafka which is working out great, but does include a broker as a middleman. This is good for us since it's designed to accumulate unconsumed data (or even roll it back) and still provide good performance - rather than having that happen on the producers or consumers.<p>It would be cool to see some stats on throughput and latency relative to # of producers / consumers and amount of data currently accumulated in the producers (since there is no middleman).

<i>This ensures that the only edge case that would result in message loss is an unclean shutdown of an nsqd process. In that case, any messages that were in memory (or any buffered writes not flushed to disk) would be lost.</i><p>I don't understand how you can call it a message delivery "guarantee" when you're susceptible to losing messages when a node dies.<p><i>One solution is to stand up redundant nsqd pairs (on separate hosts) that receive copies of the same portion of messages.</i><p>OK, delivery is only guaranteed if I run multiple independent sets of NSQd and write messages to both.<p>Regardless, it looks like an interesting project.

I love seeing how people tackle these sorts of problems. Here, the trade-off is duplicate messages reaching a client. I know others have tried a no-"ack" approach. I'm still working in the land of RabbitMQ solving all my business needs and I like it.<p>I'd love to see some numbers, reasons for decisions made, and suggested best practices for this solution other than "manual de-dupe."

It's kind of like Storm. I'd love to see a comparison written by the authors.

Hey it's awesome to see you guys tackling a larger project in Go. I've been writing some smaller services/tools with it in my day job, and I like it a lot.<p>Thanks for sharing your work.

How does this compare to ZeroMQ?

I'm too lazy to install Go just to try this (I would need to, right?) but reading through the docs this looks far more appealing to me than tent.io. Nice work, at least with respect to the general design. Flexible.

We use ActiveMQ at work and are investigating other MQs too. I dont see any info or performance graphs w.r.t message size. That would be a useful addition.

Anyone at Disqus care to weigh in on this?

Interesting if bitly guys had a look at beanstalkd before implementing nsqd.

Where is the guy with the comment about generics? Disappointing.

Does this have gob interface?