I used to work in the orbit of a distinguished eng at AWS who was famous for saying something to the effect of, "At the bottom of any scaled distributed system is either Paxos, or a bug."
Interesting reasoning! I had a similar thought about why raft doesn’t allow observers the other day.<p>I used observers with Gluster previously and went from annoying split brain scenarios to flawless clusters just by adding a few, and their resource usage was basically nothing.
Great write up! I don't have much practical experience with Paxos/Raft other than coursework. I'm curious, if I wanted to insert a row into a table, what is the overhead of all these extra nodes in the insert operation now compared to a single table?<p>I realize the answer depends on how big the cluster is, what state it is in at any given moment etc, but I'm happy to accept back of the envelope calculations/estimations!
Is there a difference between "observer" and "witness"? I'm used to the witness terminology from old literature and also from the Megastore paper, in which a <i>witness</i> votes in the Paxos election and stores the only the WAL.
I'm curious how many nodes end up in the consensus group, presumably you don't want more than 3 because throughput scales 1/N, unless their implementation can alleviate that significantly.
This is a great writeup, thanks. It will be very useful if y'all could add a post comparing neon with citusdb in terms of functionality, performance and operations.