Exactly-once Semantics: How Kafka Does it

Providing API for building applications that have transactions and help with idempotent producing is really exciting. This will help lower a lot of pains associated with building stream processing systems. Doing it with very little performance overhead is amazing.<p>I do feel that calling it &quot;Exactly-once delivery with kafka&quot; is slightly misleading as this requires the applications to be written in a certain way. The title makes is sound too general and borders on claiming something that is close to impossible. I dont want to be too critical here as the author was very honest with what this means in the blog post. Regardless of the title this is an amazing feature.

These are some pretty huge improvements to what was previously the weakest link in Kafka&#x27;s API: the Producer.<p>However, it&#x27;s important to note that this can only provide you with exactly-once semantics provided that the state&#x2F;result&#x2F;output of your consumer is itself stored in Kafka (as is the case with Kafka Streams).<p>Once you have a consumer that, for example, makes non-idempotent updates to a database, there&#x27;s the potential for duplication: if the consumer exits after updating the database, but before committing the Kafka offsets. Alternatively, it can lead to &quot;message loss&quot; if you use transactions on the database, and the application exits after the offsets were committed, but before the database transaction was committed.<p>The traditional solution to this problem is to provide your own offset storage for this consumer within the database itself, and update these offsets in the same transaction as the database modifications. However, I&#x27;m not certain that, combined with the new Producer API, this would provide exactly-once semantics.<p>Even if it doesn&#x27;t, it&#x27;s still a huge improvement, and significantly reduces the situations under which duplicate messages can be received.

There&#x27;s lots of words in this article (and design doc: <a href="http:&#x2F;&#x2F;goo.gl&#x2F;EsdTXo" rel="nofollow">http:&#x2F;&#x2F;goo.gl&#x2F;EsdTXo</a> ) but it&#x27;s missing a very simple key points?<p>1. State diagram: What are the messages exchanged between Producer and Consumer (and how many round trips to confirm a message)<p>2. At what point does each side consider the message to have been delivered?<p>3. Has this been tested empirically? (i.e: Setup producer, consumer, and partition&#x2F;kill each side randomly to see if messages get lost)<p>The one thing I don&#x27;t understand is the following. The two parties communicate by message passing. At some point the message will transition to a new state (i.e: delivered). That transition cannot happen on both sides at the same time. So how do you handle the failure of sending of the last message? Do you stage messages until after the timeout period has passed?

This is pretty cool stuff. I have been working with Apache Flink where the computational model is exactly once (which is still hugely useful) but it still comes with caveats about duplicate messages downstream, which makes writing to downstream db or having multiple flink processes tricky. Often times this is done by windowing data and producing complete snapshots of that window (i.e. you need to keep all the state for a given window) such that the message downstream is idempotent and then aggregating windows together (which can be non trivial for certain aggregations).<p>Once flink (and other systems) support this, I think it will really be a game changer. It will allow for doing things like sending an increment downstream rather than needing to keep all that state. Instead of mostly seeing stream processing as a thing we do for analytic use cases, it can really become the backbone for streaming applications.<p>Event Sourcing&#x2F;CQRS has been an idea for a while, but in practice, is difficult to do because to the inherent difficulty in dealing with message semantics and consistency (see <a href="https:&#x2F;&#x2F;data-artisans.com&#x2F;blog&#x2F;drivetribe-cqrs-apache-flink" rel="nofollow">https:&#x2F;&#x2F;data-artisans.com&#x2F;blog&#x2F;drivetribe-cqrs-apache-flink</a> for a good write up about such an app). The ability to independently optimize for both reads and writes while also not having to make all messages be idempotent, in my mind, will make this feasible for a broad range of teams and won&#x27;t require a huge amount of work in thinking up clever message formats or working through as many failure scenarios.<p>That being said, I fully expect this to bite hard when the caveats aren&#x27;t understood (such as when interacting with external databases) and there are still other hard problems. Like creating a consistent down stream view of an app in terms of business events rather than hooking into a database transaction log.<p>Still though, I think these are the sort of solutions needed to make distributed systems easier, even though there are lots of caveats :)

Boy would I love to see a Jepsen test of this one.

So its exactly once semantics, not exactly once delivery. Adding a dedup is not exactly once delivery, its being idempotent, its exactly once commit, we&#x27;ve had this for years. Having clients request for committed messages and keep track of their progress is not exactly once delivery, its exactly once request. Exactly once delivery has messages pushed to clients. As blogs have noted this is not possible.

<a href="https:&#x2F;&#x2F;techcrunch.com&#x2F;2017&#x2F;06&#x2F;30&#x2F;confluent-achieves-holy-grail-of-exactly-once-delivery-on-kafka-messaging-service&#x2F;" rel="nofollow">https:&#x2F;&#x2F;techcrunch.com&#x2F;2017&#x2F;06&#x2F;30&#x2F;confluent-achieves-holy-gr...</a><p>There is a crazy claim of &#x27;solving the unsolved problem for so many years&#x27; from Confluent CTO Neha Narkhed as mentioned below from the above article.<p>“It’s kind of insane. It’s been an open problem for so many years and Kafka has solved it — but how do we know it actually works?” she asked echoing the doubts of the community.<p>This is solved only in consumer-transform-produce scenario addressing kafka streams exactly once requriements. This is a nice feature for kafka streams but it is better to avoid tall claims like the above.<p>Actually blog post also started with major claims but it was made clear in later paragraphs on how it is acheived through de-duplicating messages in broker and the limitations on consumer side. Title of the blog post and comments in the tech crunch does not seem to be in good spirit.<p>Idempotent producer does not give any API to solve exactly once producer fecthing messages from target systems and sending them to topics.<p>I like kafka and the design choices taken to keep things simple for users but all these tall claims should have been avoided.

Does this just mean (in TL;DR form) that Kafka now has producers generate an ID for each message they send, and Kafka deduplicates it now instead of requiring deduplication on the end consumer?

Dear hacker news:<p>1. Please read the section entitled &quot;Is this Magical Pixie Dust I can sprinkle on my app?&quot; Before making angry comments. Answer: for general consumer apps just consuming messages, no. However, Kafka&#x27;s design, which let&#x27;s the consumer control it&#x27;s position in the log, combined with this feature which eliminates duplicates in the log make building end-to-end exactly once messaging using the consumer quite approachable. For stream processing using Kafka&#x27;s Streams API (<a href="https:&#x2F;&#x2F;www.confluent.io&#x2F;blog&#x2F;introducing-kafka-streams-stream-processing-made-simple&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.confluent.io&#x2F;blog&#x2F;introducing-kafka-streams-stre...</a>), where you are taking input streams, maintaining state, and producing output streams, the answer is that it actually kind of is like magic pixie dust, you change a single config and get exactly-once processing semantics on arbitrary code. Obviously you still need to get the resulting data our of Kafka, but when combined with any off-the-shelf Kafka connector which maintains exactly-once you can get this for free. So for that style of app design you actually can get correct results end-to-end without needing to do any of the hard stuff.<p>2. Someone is going to come and say &quot;FLP means that exactly once messaging is impossible!&quot; or something else from a half-understood tidbit of distributed systems theory they picked up on a blog. Let me preempt that. FLP is about the impossibility of consensus in a fully asynchronous setting (e.g. no timeout-based failure detection). Of course as you know the vast majority of the systems you use in AWS or your own datacenter depend in deep ways on consensus. Kafka itself is a CP log, about as close of a direct analog to consensus as you could ask for. Obviously Kafka and all these systems are &quot;impossible&quot; in the same sense that if you can make the network or other latency issue bad enough you can make the system unavailable for writes. This feature doesn&#x27;t change that at all, it just piggybacks on the existing consensus Kafka does. It doesn&#x27;t violate any theorems in distributed systems theory: Kafka and any consensus-based system can&#x27;t work in a fully asynchronous setting, Kafka was a CP system in the CAP sense prior to this feature and this feature doesn&#x27;t change that guarantee.<p>For those who want a deeper dive into how it all works there is a longer write up on the design here: <a href="https:&#x2F;&#x2F;cwiki.apache.org&#x2F;confluence&#x2F;display&#x2F;KAFKA&#x2F;KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging" rel="nofollow">https:&#x2F;&#x2F;cwiki.apache.org&#x2F;confluence&#x2F;display&#x2F;KAFKA&#x2F;KIP-98+-+E...</a> The use for stream processing is described here: <a href="https:&#x2F;&#x2F;cwiki.apache.org&#x2F;confluence&#x2F;display&#x2F;KAFKA&#x2F;KIP-129%3A+Streams+Exactly-Once+Semantics" rel="nofollow">https:&#x2F;&#x2F;cwiki.apache.org&#x2F;confluence&#x2F;display&#x2F;KAFKA&#x2F;KIP-129%3A...</a>

Is this similar to MQTT qos 2 ?

Sounds great, but I&#x27;ll wait until Aphyr has done his worst to prove that it doesn&#x27;t work.