How to use RabbitMQ in service integration

RabbitMQ went on my &quot;never again&quot; list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn&#x27;t many viable alternatives to AMQP that were OSS and reasonable.<p>However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of.<p>For the longest time I just made do with Kafka but this had serious deficiencies when implementing queues because of the cumulative ack only nature of Kafka.<p>Recently I have started using Pulsar which provides selective ack and all the best parts of AMQP without the complexity and unneeded parts. i.e it has things like scheduled delivery and TTLs in addition to the all important shared subscription which makes queues &quot;just work&quot; on top of streams.<p>If you want something like RabbitMQ but with a simpler API and are comfortable with JVM services give Pulsar a go. It&#x27;s not for everyone but if you are already using a lot of the big data stack it&#x27;s probably a good fit.

I&#x27;ve skimmed the comments here and of course the article. Some points based on experience with Kafka, RabbitMQ and exploratory (and ahead of hype cycle) look at Pulsar:<p>RabbitMQ is an excellent <i>messaging</i> middleware. But simply remember that it is not designed for optimal performance when holding on to data. &quot;It is a river, not a lake&quot;. Performance is very sensitive to the amount of data that is in flight between send and receive end points.<p>Kafka is a &quot;lake&quot;, but will not give you the rich routing and diverse semantics of Rabbit. If you are building &#x27;event sourced&#x27; systems, Kafka and similar systems are a better choice.<p>Pulsar has a highly articulated architecture. It is built on Bookkeeper and decouples the persistent store servers from the client servicing servers. If you want to avoid the rebalancing pain of Kafka, Pulsar is the solution. However, Puslar has many more moving pieces.<p>Both RabbitMQ and Pulsar are authentic &#x27;middle ware&#x27;, and extensible. Kafka is, true to its genesis, a highly performant <i>distributed log</i>.<p>Durable, recoverable, and performant distributed messaging&#x2F;journaling is inherently complex. Make sure you know what is it that you precisely require, and one of the above solutions will likely serve you well.

At work, we&#x27;ve been using a RabbitMQ instance as a central integration point (for about 7ish years now, I believe), and it&#x27;s been working very well so far.<p>My observations so far:<p>* Running a single RabbitMQ is pretty boring in the good sense.<p>* We haven&#x27;t managed to switch to a cluster for HA yet; it seems that software that deals with RabbitMQ clusters must be cluster aware (consume from queues on all instances and the likes), and it wasn&#x27;t worth our effort to fix all the applications.<p>* In the long run, the lack of tooling is hurting us. Want to do green-blue deployments? Canary deployments? When your services run on HTTP(S), there are simply tools for that. When your services consume from AMQP queues? You have to go searching for solutions, possibly build your own plumbing.<p>In the end, it turns out that we use publish&#x2F;subscribe far less often that direct request&#x2F;response patterns, so for new stuff I&#x27;d likely go with HTTPS instead of AMQP today.

I find that one of the major drawbacks of RabbitMQ is how much CPU it requires to run efficiently. If you add clustering on top of it it can easily become the biggest CPU hog in a containerized cluster. However it does what it says, and it does quite well.

DON&#x27;T. RabbitMQ is fragile and breaks under low memory&#x2F;high swap conditions. Usually due to epmd breaking. I know, have to fix it (usually by killing epmd) daily.

Same here in regards to liking RabbitMQ until you need to scale it. I wasted a huge amount of time years ago trying to build a HA cluster that worked when you hit it with a load.<p>What didn&#x27;t help was the pika python client had so many issues. One of my github problem reports had a 20 line demo showing how it broke with anything but a trivial load. I gave up soon after as I had further problems in other languages. Over a year later someone looked at it and said &#x27;yep fails under load&#x27; and fixed it.<p>One of those projects that showed so much promise only to make me sad.

When I proposed to use RabbitMQ for service integration my colleagues didn&#x27;t agree to go with it because it &quot;won&#x27;t scale&quot; and AWS doesn&#x27;t provide a managed service. Because of an AWS fetish and awful development experience the product won&#x27;t launch anytime soon.