ZeroMQ - Disconnects are Good for You

Yeah, don't use REQ/REP.<p>I used to be a big ZeroMQ fan, but honestly, the library trades sanity for performance and has collapsed in on itself because the code is no longer maintainable. Last I checked, it's being completely rewritten in a different programming language. Maybe the result will be fine, but the core features of retrying connections and remembering dropped messages for clients that disappear temporarily are easy enough to write yourself.<p>(I do like Mongrel2's use of that feature to allow me to make an HTTP request and <i>then</i> start the backend server. And the right place for this is a library. It's just that ZeroMQ had too many features and too much code.)

A implementation of this kind of usage pattern is already provided in the ZMQ guide:<p><a href="http://zguide.zeromq.org/page:all#Client-side-Reliability-Lazy-Pirate-Pattern" rel="nofollow">http://zguide.zeromq.org/page:all#Client-side-Reliability-La...</a><p>If the client has not received a response by the timeout it should close the connection itself and reopen a new one. Whether this is a sutible solution for the blog author issue I don't know, but work well for RPC connections with little or no state.

&#62; This could probably be improved by having a background thread that uses a ZeroMQ socket for heartbeating.<p>Don't use heartbeats on REQ/REP, because they won't work well with the lockstep communication fashion of those socket types. Also, you have to be careful because ZeroMQ sockets are not thread-safe, so the background and active thread must coordinate through a lock, or work in an implementation that handles this implicitly for you.<p>In ZeroRPC, we solve this by using XREQ/XREP with heartbeats. This has worked out pretty well in practice.

Rather than polling, zeromq &#62;= 2.2 allows you to set ZMQ_RCVTIMEO on the socket which seems to be what the author is after. It would be nice to be notified of disconnected peers, but the timeout + retry approach has been good enough for me.

Great article. ZeroMQ had a "smell" that I couldn't put my finger and thing article kind of nailed it. In retrospect I guess the smell is that it is tightly couples both sides of the network to make performance claims. It sacrifices robustness for performance.<p>I guess that it was developed for financial trading applications. Maybe it will work fine for those -- you have a few machines and high network connectivity between them. But people started doing "data center" stuff with 0MQ. Then you have geographical separation, and WAN latency and reliability.

I think the problem is people come into zeromq expecting a high level library that handles all the details, zeromq does not do that, you need to handle reliability and disconnect behavior yourself. I agree that the default behavior in this case could be saner, but it's pretty easy to build reliable request reply in many different ways as illustrated in the guide, so I'm fine with it.<p>The benefit though, is that in zeromq you get to (and are forced to) choose exactly how your messaging patterns are reliable (or not)

The better solution? That the 0mq libs do the right thing and don't get wedged. It shouldn't be on the users of the API to handle this.<p>EDITED: my point is general; it should be 0mq libs doing the timeouts and keepalives and so on and only pushing meaningful error handling like "the server has gone away and cannot reconnect" back up to the user.

It seems to be that the better solution might be just using Twisted and regular networking techniques.

Sure, the REQ/REQ sockets are limited, especially because they force the Request/Reply/Request/Reply/... series. I don't think any complex applications use this. I recently built an application using DEALER/ROUTER sockets, where you can send multiple requests, without having to wait for responses, etc. Additionally, no application should rely on receiving a response, the poller he suggests solves this problem nicely (although I don't think it necessary to wrap it into send/recv methods as pyzmq offers a nice polling API).

<i>Carries messages across inproc, IPC, TCP, and multicast.</i><p>so when you are using actual sockets across the network, it uses TCP. So ZMQ should be able to detect disconnects rather easily.

The solution I use is the one I mentioned to Armin on twitter: <a href="https://gist.github.com/2994781" rel="nofollow">https://gist.github.com/2994781</a><p>It's not really idea that a synchronous connection doesn't have notification of a connection failure, but this has been working fine for us for ages.

This badly reminds me at my MQSeries experience.<p>I wonder - is there any MQ that does not suck ?

The TCP stack takes care of this problem this is an insane attempt at POST mature optimization.