Let's consign CAP to the cabinet of curiosities

So you’re setting up a multi-region RDS. If region A goes down, do you continue to accept writes to region B?<p>A bank: No! If region A goes down, do <i>not</i> process updates in B until A is back up! We’d rather be down than wrong!<p>A web forum: Yes! We can reconcile later when A comes back up. Until then keep serving traffic!<p>CAP theorem doesn’t let you treat the cloud as a magic infinite availability box. You still have to design your system to pick the appropriate behavior <i>when</i> something breaks. No one without deep insight into your business needs can decide for you, either. You’re on the hook for choosing.

You wish.<p>&gt; DNS, multi-cast, or some other mechanism directs them towards a healthy load balancer on the healthy side of the partition<p>Incidentally that&#x27;s where CAP makes it&#x27;s appearance and bites your ass.<p>No amount of VRRP, UCARP wishful thinking can guarantee a conclusion on what partition is &quot;correct&quot; in presence of a network partition between load balancer nodes.<p>Also, who determines where to point the DNS? A single point of failure VPS? Or perhaps a group of distributed machines voting? Yeah.<p>You still need to perform the analysis. It&#x27;s just that some cloud providers offer the distributed voting clusters as a service and take care of the DNS and load balancer switchover for you.<p>And that&#x27;s still not enough, because you might not want to allow stragglers write to orphan databases before the whole network fencing kicks in.

I once lost an entire Christmas vacation to fixing up the damage caused when an Elasticsearch cluster running in AWS responded poorly to a network partition event and started producing results that ruined our users&#x27; day (and business records) in a &quot;costing millions of dollars&quot; kind of way.<p>It was a very old version of ES, and the specific behavior that led to the problem has been fixed for a long time now. But still, the fact that something like this can happen in a cloud deployment demonstrates that this article&#x27;s advice rests on an egregiously simplistic perspective on the possible failure modes of distributed systems.<p>In particular, the major premise that intermittent connectivity is only a problem on internetworks is just plain wrong. Hubs and switches flake out. Loose wires get jiggled. Subnetworks get congested.<p>And if you&#x27;re on the cloud, nobody even tries to pretend that they&#x27;ll tell you when server and equipment maintenance is going to happen.

When I design systems I just think about tiny traitor generals and their sneaky traitor messengers racing in the war, their clocks are broken, and some of them are deaf, blind or both.<p>CAP or no CAP, chaos will reign.<p>I think FLP (<a href="https:&#x2F;&#x2F;groups.csail.mit.edu&#x2F;tds&#x2F;papers&#x2F;Lynch&#x2F;jacm85.pdf" rel="nofollow">https:&#x2F;&#x2F;groups.csail.mit.edu&#x2F;tds&#x2F;papers&#x2F;Lynch&#x2F;jacm85.pdf</a>) is better way to think about systems.<p>I think CAP is not as relevant in the cloud because the complexity is so high that nobody even knows what is going on, so the just C part, regardless of the other letters, is ridiculously difficult even on a single computer. A book can be written just to explain write(2)&#x27;s surprise attacks.<p>So you think you have guarantees whatever the designers said they have AP or CP, and yet.. the impossible will happen twice a day (and 3 times at night when its your on-call).

The military lives in this world and will likely encourage people to continue thinking about it. Think about wearables on a submarine, as an example. Does the captain want to know his crew is fatigued, about to get sick, getting less exercise than they did on their last deployment? Yes. Can you talk to a cloud? No. Does the Admiral in Hawaii want to know those same answers about that boat, and every boat in the Group, eventually? Yes. For this situation, datacenter-aware databases are great. There are other solutions for other problems.

&gt; The CAP Theorem is Irrelevant<p>Just sprinkle the magic &quot;cloud&quot; powder on your system and ignore all the theory.<p><a href="https:&#x2F;&#x2F;ferd.ca&#x2F;beating-the-cap-theorem-checklist.html" rel="nofollow">https:&#x2F;&#x2F;ferd.ca&#x2F;beating-the-cap-theorem-checklist.html</a><p>Let&#x27;s see, let&#x27;s pick some checkboxes.<p>(x) you pushed the actual problem to another layer of the system<p>(x) you&#x27;re actually building an AP system

There&#x27;s a better rebuttal(*) of CAP in Kleppmann&#x27;s DDIA, under the title, &quot;The unhelpful CAP theorem&quot;.<p>I won&#x27;t plagiarize his text, instead the chapter references his blogpost, &quot;Please stop calling databases CP or AP&quot;: <a href="https:&#x2F;&#x2F;martin.kleppmann.com&#x2F;2015&#x2F;05&#x2F;11&#x2F;please-stop-calling-databases-cp-or-ap.html" rel="nofollow">https:&#x2F;&#x2F;martin.kleppmann.com&#x2F;2015&#x2F;05&#x2F;11&#x2F;please-stop-calling-...</a><p>(*): rebuttal I think is the wrong word, but I couldn&#x27;t think of better.

Plot twist: in the article drawings, replica one and two are split by network, and it could fail.<p>The author seems to not understand what the meaning of the P in CAP

Someone else tok ownership of the problem for you and sells you their solution : &quot;The theoretical issue is irrelevant to me&quot;.<p>Sure. Also, there&#x27;s a long list of other things that are probably irrelevant to you. That is, until your provider fails and you need to understand the situation in order to provide a workaround.<p>And slapping &quot;load-balancers&quot; everywhere on your schema is not really a solution, because load-balancers themselves are a distributed system with a state and are subject to CAP, as presented in the schema.<p>&gt; DNS, multi-cast, or some other mechanism directs them towards a healthy load balancer on the healthy side of the partition.<p>&quot;Somehow, something somewhere will fix my shit hopefully&quot;. Also, as a sidenote, a few friends would angrily shake their &quot;it&#x27;s always DNS&quot; cup reading this.<p>edit: reading the rest of the blog and author&#x27;s bio, I&#x27;m unsure whether the author is genuinely mistaken, or whether they&#x27;re advertising their employer&#x27;s product.

&gt; None of the clients need to be aware that a network partition exists (except a small number who may see their connection to the bad side drop, and be replaced by a connection to the good side).<p>What a convenient world where the client is not affected by the network partition.

As someone who&#x27;s worked extensively on distributed systems, including at a cloud provider, after reading this I think the author doesn&#x27;t actually understand the CAP theorem or the two generals problem. Their conclusions are essentially utterly incorrect.

Many things can be solved by the SEP Field[0]<p>[0]: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Somebody_else&#x27;s_problem#Douglas_Adams&#x27;_SEP_field" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Somebody_else&#x27;s_problem#Dougla...</a>

The CAP theorem is quantum mechanics of software with C*A = O(1) in theory, similarly to uncertainty principle, but in many use cases this value is so small that &quot;classical&quot; expectations of both C and A are fine.

&gt; In practice, the redundant nature of connectivity and ability to use routing mechanisms to send clients to the healthy side of partitions<p>Iow: You can have CAP as long as you can communicate across &quot;partitions&quot;.

So glad to see that the CAP &quot;theorem&quot; is being recognized as a harmful selfish meme like Fielding&#x27;s REST paper with a deadly seductive power against the overly pedantic.

I think every couple of months there&#x27;s yet another article saying the CAP theorem is irrelevant. The problem with these is that they ignore the fact that CAP theorem isn&#x27;t a guide, a framework or anything else.<p>It&#x27;s simply the formalization of a fact, and whether or not that fact is *important* (although still a fact) depends on the actual use case. Hell, it applies even to services within the same memory space, although obviously the probability of losing any of the three is orders of magnitude less than on a network.<p>Can we please move on?

Hmm this article seems misleading. I suppose it&#x27;s trying to make the point that application designers usually don&#x27;t need to think too hard about it, because it&#x27;s already being addressed by a quorum consensus protocol implemented by someone else. This is a bit of a tautology though; the author seems to be saying &#x27;assume you have a solution to CAP theorem -- now isn&#x27;t it silly to worry about CAP theorem?&#x27;.<p>One of the fundamental assumptions of CAP theorem is that you can&#x27;t tell whether or not you have a partition. If you have an oracle that can instantaneously tell you the state of every subsystem, then yeah, CAP is pointless.<p>But if one of your DBs is connected, reporting itself as alive, and throwing all its writes into &#x2F;dev&#x2F;null, you won&#x27;t be able to route traffic to a quorum of healthy instances because it&#x27;s not possible to be certain that they&#x27;re all healthy.<p>This is what CAP theorem is about: managing data in a distributed system where the status of any given system is fundamentally unknowable because of the Two Generals&#x27; Problem (<a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Two_Generals&#x27;_Problem)" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Two_Generals&#x27;_Problem)</a><p>In many cases in Cloud though, we can skip that technical stuff and design systems as if we really _did_ have an oracle that could instantaneously and perfectly tell us the state of the system, and things will typically work fine.

The point trying to be made is that with nimble infrastructure the A in CAP can be designed around to such a small amount you may as well be a CP system unless you have a really good reason to go after that 0.005% of availability. Not being CP means sacrificing the wonderful benefits that being consistent (linearizability, sequential consistency, strict serializibility) make possible. It&#x27;s hard to disagree with that sentiment, and is likely why the Local First ideology is centered on data ownership rather than that extra 0.0005 ounces of availability. Once availability is no longer the center of attention the design space can be focused on durability or latency: how many copies to read&#x2F;write before acking.<p>Unfortunately the point is lost because of the usage of the word &quot;cloud&quot;, a somewhat contrived example of solving problems by reconfiguring load balancers (in the real world certain outages might not let you reconfigure!), and missing empathy that you can&#x27;t tell people not to care about how the semantics that thinking about, or not thinking about, availability imposes on the correctness of their applications.<p>As for the usage of the word cloud: I don&#x27;t know when a set of machines becomes a cloud. Is it the APIs for management? Or when you have two or more implementations of consensus running on the set of machines?

He&#x27;s saying that you don&#x27;t need Partition Tolerance because network is never actually Partitioned. This is exactly why the Internet and the US Interstate Highway system were invented in the first place.<p>Or he&#x27;s saying you don&#x27;t need Consistency because your system isn&#x27;t actually distributed; it&#x27;s just a centralized system with hot backups.<p>It&#x27;s unclear what he&#x27;s trying to say.<p>No idea why he wrote the blog post. It doesn&#x27;t increase my confidence in the engineering equality of his employer AWS

&gt; If the partition extended to the whole big internet that clients are on, this wouldn’t work. But they typically don’t.<p>This is the key, that network partitions either keep some clients from accessing any servers, or they keep some servers from talking to each other. The former case is uninteresting because nothing can be done server-side about it. The latter is interesting and we can fix it with load balancers.<p>This conflicts with the picture painted earlier in TFA where the unhappy client is somehow stuck with the unhappy server, but let&#x27;s consider that just didactic.<p>We can also not use load balancers but have the clients talk to all the servers they can reach, when we trust the clients to behave correctly. Some architectures do this, like Lustre, which is why I mention it.<p>I see several comments here that seem to take TFA as saying that distributed consensus algorithms&#x2F;protocols are not needed, but TFA does not say that. TFA says you can have consistency, availability, and partition tolerance because network partitions between servers typically don&#x27;t extend to clients, and you can have enough servers to maintain quorum for all clients (if a quorum is not available it&#x27;s as if the whole cloud is down, then it&#x27;s not available to any clients). That is a very reasonable assertion, IMO.

So basically this is saying that the CAP theorem is irrelevant because a partition is not really have a partition (since the load balancer still can reach everybody). Hmm...<p>I agree that in modern data centers the CAP theorem is essentially irrelevant for <i>intra-DC</i> services, due the uptime and redundancy of networking H&#x2F;W (making a partition less likely than other systemic failures).<p>Across DCs I&#x27;ll claim it is still absolutely relevant.

The only concrete solution the article proposes that I can think of: Spanner uses quorum to maintain availability and consistency. Your &quot;master&quot; is TrueTime, which is considered reliable enough. You have replicated app backends. If this isn&#x27;t too generous, let&#x27;s also say the cloud handles load balancing well enough. CAP isn&#x27;t violated, but you might say the user no longer worries about it.<p>Most databases don&#x27;t work like Spanner, and Spanner has its downsides, two of them being cost and performance. So most of the time, you&#x27;re using a traditional DB with maybe a RW replica, which will sacrifice significant consistency or availability depending on whether you choose sync or async mode. And you&#x27;re back to worrying about CAP.

Weird article. Different users have different priorities and that’s what the CAP theorem expresses. The article also pretends that there’s a magic “load balancer” in the cloud that always works and also knows which segment of a partitioned network is the “correct” one (one of the points of CAP is that there’s not necessarily a “correct” side), and that no users will ever be on the “wrong” side of the partition. And not only that but all replicas see the exact same network partition. None of this is reality.<p>But the gist, I guess, is that for most applications it’s not actually that important, and that’s probably true. But when it is important, “the cloud” is not going to save you.

CAP was never designed as an end all template you blindly apply to large scale systems. Think of it more as a mental starting pointing, that systems have these trade offs you need to consider. Each system you integrate has complex and nuanced requirements that don&#x27;t neatly fall into clean buckets.<p>As always Kleppmann has a great and deep answer for this.<p><a href="https:&#x2F;&#x2F;martin.kleppmann.com&#x2F;2015&#x2F;05&#x2F;11&#x2F;please-stop-calling-databases-cp-or-ap.html" rel="nofollow">https:&#x2F;&#x2F;martin.kleppmann.com&#x2F;2015&#x2F;05&#x2F;11&#x2F;please-stop-calling-...</a>

I suspect the CAP theorem factored into the design of these cloud architectures, in such a way that it now seems irrelevant. But it probably was relevant in preventing a lot of other more complex designs.

I kinda see what the author is getting at, but I don&#x27;t buy the argument.<p>However, in the example with the network partition, it relies on proper monitoring to work out if the DB its attached to is currently in partition.<p>managing reads is a piece of piss, mostly. Its when you need propagate write to the rest of the DB system, thats where stuff gets hairy.<p>Now, most places can run from a single DB, especially as disks are fucking fast now. so CAP is never really that much of a problem. However when you go multi-region, thats when it gets interesting.

This only addresses one kind of partition.<p>What if your servers can&#x27;t talk to each other, but clients can?<p>What if clients can&#x27;t connect to any of your servers?<p>What if there are multiple partitons, and none of them have a quorum?<p>Also, changing the routing isn&#x27;t instantaneous, so you will have some period of unavailability between when the partition happens, and when the client is redirected to the partition with the quorum.

&gt; if a quorum of replicas is available to the client, they can still get both strong consistency, and uncompromised availability.<p>Then it’s not a partition.

This is just saying because the cloud system hides the implications of the theorem from you, it&#x27;s not relevant.<p>I suppose it&#x27;s kinda true in the sense that how to operate a power plant is not relevant when I turn on my lights.

This article assumes P(artitions) don&#x27;t happen, and then concludes you can have both C and A. Congrats, that&#x27;s the CAP theorem.

&gt; The formalized CAP theorem would call this system unavailable, based on their definition of availability:<p>Umm, no? That’s a picture of a partition. The partition is not able to make progress because the system is not partition tolerant. If it did it wouldn’t be consistent. It’s still available.

See also:<p>&gt; <i>In database theory, the PACELC theorem is an extension to the CAP theorem. It states that in case of network partitioning (P) in a distributed computer system, one has to choose between availability (A) and consistency (C) (as per the CAP theorem), but else (E), even when the system is running normally in the absence of partitions, one has to choose between latency (L) and loss of consistency (C).</i><p>* <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;PACELC_theorem" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;PACELC_theorem</a>

This seems like what I&#x27;ve noticed on MPP systems (a little before cloud): data replicas give a lot more availability than the number of partition events would suggest.<p>I likely need to read the paper linked, but it&#x27;s common to have an MPP database lose a node but maintain data availability. CAP applies at various levels, but the notion of availability differs:<p>1. all nodes available 2. all data available<p>Redundancy can make #2 a lot more common than #1.

The CAP theorem is irrelevant if your acceptable response time is greater than the time it takes your partitions to sync.<p>At that point you get all 3: consistency,availability, partitioning.<p>In my opinion it should be the CAPR theorem.

If you don&#x27;t care about costs...

All models are wrong, some are useful. CAP is probably at least as useful as Newtonian mechanics WHEN you are explaining why you just did a bunch of… extra stuff.<p>I would like to violate CAP, please. I would like to be nearish to c, please.<p>Here is my passport. I have done the work.

&gt; The point of this post isn’t merely to be the ten billionth blog post on the CAP theorem. It’s to issue a challenge. A request. Please, if you’re an experienced distributed systems person who’s teaching some new folks about trade-offs in your space, don’t start with CAP.<p>Yeah… no. Just because the cloud offers primitives that allow you to skip many of the challenges that the CAP theorem outlines, doesn’t mean it’s not a critical step to learning about and building novel distributed systems.<p>I think the author is confusing systems practitioners with distributed systems researchers.<p>I agree in some part, the former rarely needs to think about CAP for the majority of B2B cloud SaaS. For the latter, it seems entirely incorrect to skip CAP theorem fundamentals in one’s education.<p>tl;dr — just because Kubernetes (et al.) make building distributed systems easier, it doesn’t mean you should avoid the CAP theorem in teaching or disregard it altogether.

Every time someone tries to deprecate the nice and simple CAP theorem, it grows stronger. It&#x27;s an unstoppable freight train at this point, like the concept of relational DBs after the NoSQL fad.