TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

You Cannot Have Exactly-Once Delivery

113 pointsby tylertreatabout 10 years ago

17 comments

brianpgordonabout 10 years ago
Well, you have have exactly-once delivery unless there are network partitions, which is not a particularly surprising limitation. As the author admits, it takes considerable cleverness in the implementation details in order to achieve that abstraction, but plenty of tools have done it.<p>&quot;People often bend the meaning of “delivery” in order to make their system fit the semantics of exactly-once, or in other cases, the term is overloaded to mean something entirely different. State-machine replication is a good example of this. Atomic broadcast protocols ensure messages are delivered reliably and in order. The truth is, we can’t deliver messages reliably and in order in the face of network partitions and crashes without a high degree of coordination. This coordination, of course, comes at a cost (latency and availability), while still relying on at-least-once semantics.&quot;<p>This reminds me of the problem of reliable data transfer over an unreliable network. It&#x27;s theoretically impossible, but TCP is still a practical, useful abstraction.
评论 #9268500 未加载
Animatsabout 10 years ago
Sure you can. What you can&#x27;t have is, after a loss of communication, unambiguous knowledge about whether the other end got the last message.<p>When communication is reestablished, that issue can be sorted out. See &quot;two-phase commit&quot;.
评论 #9267812 未加载
评论 #9267781 未加载
评论 #9268239 未加载
fskabout 10 years ago
Then how does Domino&#x27;s get me a pizza within 30 minutes?
评论 #9267880 未加载
评论 #9268940 未加载
评论 #9267587 未加载
评论 #9267904 未加载
zhengabout 10 years ago
Most systems I know which require this just do at-least once on the sending side and dedupe on the receiving side. If you build this into your framework, to applications you have exactly-once barring unbounded partitions and the like.
评论 #9267693 未加载
jacques_chesterabout 10 years ago
My understanding of the FLP result is that it only applies to algorithms without timeouts.<p>Because it restates that, without timeouts, in theory, you could be waiting forever for a message to arrive. It&#x27;s formally proved and whatnot, but it&#x27;s not a super-surprising result when you restate it in common terms.<p>For message delivery, it seems that FLP says that you need to give up at-least-once, because at-least-once ostensibly requires an infinite number of retries. I&#x27;m not sure how it would apply to at-most-once.<p>Exactly-once is, so far as I can tell, a bit of a strawman. But it&#x27;s a strawman that we all fall for, the first time we break stuff into different systems.<p>As a disclaimer, I am still wrapping my head around this stuff. Don&#x27;t rely on me. Not even if your name is Salvatore.
评论 #9266984 未加载
评论 #9267235 未加载
zkhaliqueabout 10 years ago
Is it just me or did the following alliteration catch someone&#x27;s eye?<p>DD: I myself shared many of these misconceptions, so I try not to <i>demean or dismiss</i><p>EE: but rather <i>educate and enlighten</i>, hopefully while sounding less preachy than that just did.<p>FF: I continue to learn only by <i>following in the footsteps</i> of others.
评论 #9267646 未加载
评论 #9267922 未加载
eclarkabout 10 years ago
No you can&#x27;t have exactly once delivery. However you can mitigate this if your datastore for your output from message processing is the same as your datastore for queue. With that and atomic mutations (de-queue and return result), it does allow practical solutions for almost all edge cases.
jermoabout 10 years ago
Another great read on this subject &quot;Exactly-Once Delivery May Not Be What You Want&quot; <a href="https://brooker.co.za/blog/2014/11/15/exactly-once.html" rel="nofollow">https:&#x2F;&#x2F;brooker.co.za&#x2F;blog&#x2F;2014&#x2F;11&#x2F;15&#x2F;exactly-once.html</a>
peterwwillisabout 10 years ago
I hope the author isn&#x27;t making an argument that network partitions are somehow the only consideration of whether a message is delivered. Even on a single user single process machine, delivery can&#x27;t be totally guaranteed.&#x27;exactly once&#x27; delivery sounds like &#x27;guaranteed perfect one time delivery under all circumstances&#x27;, which is a dumb way to rephrase an idea whose literal translation is to deliver something one time and not more or less.<p>In other words, of course you can deliver something exactly one time. Just not every time.
billpgabout 10 years ago
Shameless plug:<p>I wrote about this issue with a lot of APIs some time ago. When a connection is broken unexpectedly, the protocol simply has no way to recover the state of play and there&#x27;s a risk of duplicate transactions popping up.<p><a href="http://blog.hackensplat.com/2014/07/is-your-api-broken.html" rel="nofollow">http:&#x2F;&#x2F;blog.hackensplat.com&#x2F;2014&#x2F;07&#x2F;is-your-api-broken.html</a>
rusanuabout 10 years ago
Having spent 7 year of my life implementing an Exactly-Once-In-Order (EOIO) messaging system in SQL Server Service Broker[0] (SSB), I take somehow exception to the author claim.<p>Here is how SSB achieves EOIO:<p>- initiator establishes intent to communicate using BEGIN DIALOG[1] statement (SSB dialogs are the equivalent of a durable, long lived, TCP session). This creates the necessary state in the database by creating a row in sys.conversation_endpoints, with initial send_sequence_number 0.<p>- sender issues SEND[2] statement. The message is assigned the current send_sequence_number (0), the sys.conversation_endpoint send_sequence_number is incremented to 1, and the message is inserted in sys.transmission_queue<p>- after commit of SEND transaction a background transmitter reads the message from sys.transmission_queue, connects to destination, delivers the message over wire (TCP).<p>- target reconstructs message from wire, in a single transaction creates a row in it sys.conversation_endpoints (receive_sequence_number is 0) and delivers the message into the destination queue<p>- after commit of the message delivery, the target constructs an acknowledgement message and sends it back to initiator over the wire<p>- the sender gets the acknowledgement of message 0 and deletes the message from sys.transmission_queue<p>- sender may retry delivery periodically if it does not receive the ack<p>- target will send back an ack immediately on receipt of a duplicate (message sequence number is less than current receive_sequence_number)<p>What this protocol achieves is idempotency of delivery, hidden from the messaging application. Database WAL ensures stability in presence of crashes. Eg. if the target crashes in the middle of enqueueing the message into destination queue then the entire partial processing of the enqueue is rolled back on recovery and next retry from sender will succeed. If target crashes after processing is committed but before sending the ack then on recovery the enqueue is successful and the next retry from initiator will immediately send ack an ack, allowing initiator to delete the retried message and make progress (send the next message in sequence). Note that there is no two-phase-commit involved.<p>Retries of unacknowledged messages occur for the dialog lifetime, which can be days, months, even years. Databases are good at keeping state for so long. SSB uses logical names for destination (from &#x27;foo&#x27; to &#x27;bar&#x27;) and routing can be reconfigured mid-flight (ie. the location where &#x27;bar&#x27; is hosted can be changed transparently). Long lived state and routing allow for transparent reconfiguraiton of network topologies, handle downtime, manage disaster (target is lost and rebuild from backups). Most of the time this is transparent to the SSB application.<p>Furthermore, the guarantees can be extended to application semantics as well. Applications dequeue messages using RECEIVE[3] statement. In a single transaction the application would issue a RECEIVE to dequeue the next available message, lookup app state perteining the message, modify the state, send a response using SEND[2], commit. Again WAL guarantees consistency, after a crash everything is rolled back and the application would go again through exactly the same sequence (the response SEND cannot communicate anything on the wire until after commit, see above).<p>So EOIO is possible.<p>One has to understand the trade offs implied. Something like SSB will trade off latency for durability. Applications need not worry about retries, duplicates, missing messages, routing etc as long as they are capable of handling responses comming back <i>hours</i> (or maybe <i>weeks</i>) after the request was sent. And application processing of a message is idempotent (RECEIVE -&gt; process -&gt; crash -&gt; rollback -&gt; RECEIVE -&gt; re-process) only as long as the processing is entirely database bound (update states in some app tables, not make REST calls). Yet such apps are not unusual: they use some database to store state and communicate with some other app that also uses a database to store state. Unlike most messaging systems, SSB stores the messages <i>in</i> the database thus achiving WAL consitency along with the app state. Many SSB applications are entirely contained in the database, the code itself is contained. They use SSB activation [4] to react to incoming message, without keeping any state in memory.<p>In SSB both the initiator and the sender are monolitic, SMP systems (not distributed). Together the two form a distributed system. It trades of availability over partitioning, but one has to understand how this trade off occurs. In case of parittioning (target is unreacheable) the application continues to be available locally (the SEND statement succeeds). If the netowrk paritioning is not resolved <i>over the lifetime of the dialog</i>, then the applicaiton will see an error. If the paritioning is resolved then message flows resumes and the applicaiton layer responses start showing up in the queue. Again, activation and durable state make this easy to handle, as long as a latency of potentially <i>days</i> makes sense in the business. Shorter lifetimes (hours, minutes) are certainly possible and in such cases, if network partitioning is not resolved in time, the timeout error will occur sooner.<p><pre><code> [0] https:&#x2F;&#x2F;msdn.microsoft.com&#x2F;en-us&#x2F;library&#x2F;bb522893.aspx [1] https:&#x2F;&#x2F;msdn.microsoft.com&#x2F;en-us&#x2F;library&#x2F;ms187377.aspx [2] https:&#x2F;&#x2F;msdn.microsoft.com&#x2F;en-us&#x2F;library&#x2F;ms188407.aspx [3] https:&#x2F;&#x2F;msdn.microsoft.com&#x2F;en-us&#x2F;library&#x2F;ms186963.aspx [4] https:&#x2F;&#x2F;technet.microsoft.com&#x2F;en-us&#x2F;library&#x2F;ms171617.aspx</code></pre>
ryanjshawabout 10 years ago
This topic seems to come up regularly, but I feel the discussions I see here are far too heavy to digest for the people who would benefit most from understanding the issues being presented (OP post is &gt; 1,300 words).<p>I believe what it really comes down to is that people new to distributed processing think they want &quot;exactly once delivery&quot;, but later they (hopefully) learn they really want &quot;exactly once processing&quot;. For example:<p>&gt; We send him a series of text messages with turn-by-turn directions, but one of the messages is delivered twice! Our friend isn’t too happy when he finds himself in the bad part of town.<p>This is easily resolved by amending each message with an ordinal indicator, e.g. &quot;1. Head towards the McD&quot;, &quot;2. Turn left at the traffic lights&quot;, etc. The receiver then dedupes and the instructions follow &quot;exactly once processing&quot;. Processing messages exactly once is a &quot;business rule&quot; and the responsibility for doing so lies in the business logic of the application.<p>This example also brings up another typical point of confusion in building distributed systems: people actually want &quot;ordered processing&quot;, not &quot;ordered delivery&quot;. The physical receiving order does not matter: your friend will not attempt to follow instruction #2 without first following instruction #1. If instruction #2 is received first, your friend will wait for instruction #1.<p>It&#x27;s also important to note that the desired processing order of the messages has nothing to do with the physical sending order: I could be receiving directions to two different places from two different people, and it doesn&#x27;t matter what order they are sent or received, just that all the messages get to me! A great article covering these topics in more detail with other real world examples is &quot;Nobody Needs Reliable Messaging&quot; [1].<p>I think it is useful to try understand why new distributed system builders run into these difficulties. I suspect they try to apply local procedure call semantics to distributed processing (a fool&#x27;s errand), and message queue middleware works well enough that a naive &quot;fire and forget&quot; approach is the first strategy they attempt. When they subsequently lose their first message (hopefully before going into production), it&#x27;s natural to think in terms of patching (e.g. distributed transactions, confirmation protocols, etc.) rather than to consider if the overall design pattern is appropriate.<p>Oddly, there is at least one very well designed solution that addresses these challenges - the Data Distribution Service (DDS) [2] - but I almost never see or hear about it at any of my clients.<p>[1] <a href="http://www.infoq.com/articles/no-reliable-messaging" rel="nofollow">http:&#x2F;&#x2F;www.infoq.com&#x2F;articles&#x2F;no-reliable-messaging</a> [2] <a href="http://portals.omg.org/dds/" rel="nofollow">http:&#x2F;&#x2F;portals.omg.org&#x2F;dds&#x2F;</a>
评论 #9267545 未加载
einarvollsetabout 10 years ago
But you can have it with probability 1...
dbenhurabout 10 years ago
Life gets more pleasant when you internalize that consistency is not aligned with how the universe works and idempotence is your friend.
jv22222about 10 years ago
Could the blockchain be used as a send only once system?<p>I mean if you used the blockchain as a message queue it could be pretty reliable, no?
评论 #9268080 未加载
评论 #9267827 未加载
williamcottonabout 10 years ago
Would a system that prevents double-spending of a digital currency be equivalent with exactly-once delivery?
评论 #9272459 未加载
coopsabout 10 years ago
The best you can do is two-phase commit.
评论 #9267473 未加载