On the one hand, I've seen people (including myself) try to hack job-queue like semantics onto Kafka many a time, and it always hits issues once redelivery or backoff comes up. So it's nice to see them considering making this a first-class citizen of Kafka.<p>On the other hand, Kafka isn't the only player in the queue game nowadays. If you need message queue and job queue semantics combined (which you likely do), just use Pulsar.
It's looking like PostgreSQL vs MongoDB...<p>RabbitMQ has implemented Streams and "Super Streams":<p>> Super streams are a way to scale out by partitioning a large stream into smaller streams. They integrate with single active consumer to preserve message order within a partition. Super streams are available starting with RabbitMQ 3.11.<p><a href="https://www.rabbitmq.com/streams.html" rel="nofollow noreferrer">https://www.rabbitmq.com/streams.html</a>
The lack of Queue like support as a first class citizen in Kafka has kept a few things on the sidelines for me.<p>Native capacity to do queueing is exciting, especially the concept of share groups to allow potentially different types of queues and shares in the future.<p>It’s might not be appropriate, but one step closer to eating more workflow engines for lunch.
i'm having a hard time understanding what scenario requires a queue semantic instead of a stream one.<p>Is is the ability to parallelize consuming to a super large amount, without having to setup partitions ?<p>I'm planning on using kafka for a job queue, and i like the idea that i can add ordering definition if i need to, and that i can keep the jobs in the queue for auditing later on if i need to. What am i missing ?