Interesting footnote:<p>> In the 2022 engagement, the client’s engineers were enthusiastic about the prospect of a public analysis, and Jepsen was allowed to file public issues against systems including etcd. Following the conclusion of the contract, Jepsen independently completed a written report discussing the behaviors we’d found in etcd. However, Jepsen was unable to secure official permission from the client’s legal department to disclose that the client had funded part of the work. This created an unusual state of affairs: the issues, test suite, and reproduction instructions were all public, but per Jepsen’s ethics policy, the analysis itself could not be published. Jepsen shelved that analysis and it remains unpublished. The present analysis is based on entirely new work and verifies a different software system: jetcd, rather than etcd
Your posts are something I have in my bookmarks and reference regularly as I continue to build my own distributed data system. Thanks for continuing to test and report on these issues. These posts have clarified a lot of details about the consistency guarantees of these systems that I really couldn’t discern from their own documentation. The knowledge is invaluable with how developers lean towards just trusting the system they consume to be correct.
The first bug is a great reminder that even strict serializability doesn't imply idempotency. If you're doing non-idempotent operations like unconditional writes, you've got to think very carefully before you add any retries to a system. Even with conditional writes, you need to think carefully about ABA bugs.<p>Both of these bugs are a great reminder that distributed system behavior includes clients. From the application's perspective bugs like this being introduced by the client isn't any practically different from them being introduced by the server - the same badness happens. A database needs to consider it's properties end-to-end from the application API.<p>It's also a great reminder that APIs that make it hard for clients to do the right thing will likely lead to bugs like this. Failures happen, and a good API needs to be designed in a way that allows the client to do something sensible following a failure. A great API makes it easy for a client to do something sensible, and hard for a client to do the wrong thing. Perhaps my favorite non-distributed example of this is AES-GCM, the ubiquitous AEAD crypto primitive: one tiny bug (reusing an IV) completely blows up the whole scheme.<p>And, as always, this is great stuff from Kyle. His Jepsen work has been moving the industry forward for years, and it's great to see him continue it (and continue to put the effort into writing up his findings so clearly).