Kafka vs. Redpanda performance – do the claims add up?

alex here, original author of redpanda<p>is hard to respond to a 6-part blog series content - released all at once - on an HN thread.<p>- what we can deterministically show is data loss on apache kafka with no fsync() [shouldn&#x27;t be a surprise to anyone] - stay tuned for an update here.<p>- the kafka partition model of one segment per partition could be optimized in both arch<p>- the benefit for all of us, is that all of these things will be committed to the OMB (open messaging benchmark) and will be on git for anyone interested in running it themselves.<p>- we welcome all confluent customers (since the post is from the field cto office) to benchmark against us and choose the best platform. this is how engineering is done. In fact, we will help you run it for you at no cost. Your hardware, your workload head-to-head. We&#x27;ll help you set it up with both.... but let&#x27;s keep the rest of the thread technical.<p>- log.flush.interval.messages=1 - this is something we&#x27;ve taken a stance a long long time ago in 2019. As someone who has personally talked to hundreds of enterprises to date, most workloads in the world should err on the side of safety and flushing to disk (fsync()). Hardware is very good today and you no longer have to choose between safety and reasonable performance. This isn&#x27;t the high latency you used to see on spinning disks.

This was a nice read! There are a few issues on both sides, some that others have mentioned and some that I have not seen yet:<p>For Redpanda:<p>1. I don&#x27;t like that they did not include full disk performance, not sure if that was intentional but it feels like it... Seems like and obvious gap in their testing. Perhaps most of their workloads have records time out rather than get pushed out by bytes first, not sure.<p>2. Their benchmark was def selective, sure, but they sell via proof of performance for tested workloads IIUC, no via their posted benchmarks. The posted benchmarks just get them into the proof stage in a sales pipeline.<p>For Kafka (and Confluent, and this test):<p>1. Don&#x27;t turn off fsync for Kafka if you leave it on with Redpanda, that&#x27;s certainly not a fair test.<p>Batching should be done on the client side anyway, as most packages already do by default. If you are worried about too many fsyncs degrading performance, batch harder on your clients. It&#x27;s the better way to batch anyway.<p>2. If confluent cloud is using java 11, then I don&#x27;t like that java 17 is used for this either. It&#x27;s not a fair comparison seeing that most people will want it managed anyways, so it gives unrealistic expectations of what they can get<p>3. Confluent charges a stupid amount of money<p>4. The author works for Confluent, so I&#x27;m not convinced that this test would have been posted if they saw Redpanda greatly outperform Kafka<p>With Both:<p>1. Exactly once delivery is total marketing BS. At least Redpanda mentions you need idempotency, but you get exactly once behavior with full idempotency anyway. What you build should be prepared for this, not the infra you use IMO as all you need is one external system to break this promise for the whole system to lose it<p>I prefer Redpanda as I find it easier to run, and Redpanda actually cares about their users whether they are paid or not. Confluent wont talk to you unless you have a monthly budget of at least $10k, Redpanda has extremely helpful people in their slack just waiting to talk to you.<p>Ultimately you don&#x27;t just buy into the software, you buy into the team backing it, and I&#x27;d pick Redpanda easily, knowing that they can actually help me and care without needing to give them $10k.

&gt;<i>Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety.</i><p>Respect to the Kafka team as Kafka is an incredible piece of software, but the Mongo guys got torched for eternity for pulling the same shenanigans.

&gt; Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety.<p>And then in this article it&#x27;s explained how Kafka is actually unsafe:<p>&gt; Kafka may handle simultaneous broker crashes but simultaneous power failure is a problem.<p>just against simultaneous node crashes (whole VM&#x2F;machine).<p>I mean - sure in practice running in different AZs, etc. will probably be good enough, but technically...

&gt; Redpanda end-to-end latency of their 1 GB&#x2F;s benchmark increased by a large amount once the brokers reached their data retention limit and started deleting segment files. Current benchmarks are based on empty drive performance.<p>This seems really disingenuous to use empty drive performance, since anyone who cares about performance is going to be caring about continuous use.

I&#x27;d like to see these on OpenJDK 11, since that&#x27;s what Confluent is running on and the author makes a point of switching to 17 even though he works for Confluent.<p>In either case, Confluent Platform is ridiculously expensive and approached the costs (licensing alone) for our entire cloud spend. I&#x27;d love to see more run-on-k8s alternatives to CFK.

We really wanted to try redpanda, but operationally it does not appear to be very k8s* native and infact looks like a lot of one off hand holding to get it working properly.<p>Hopefully that can get ironed out in the future. Until then we will stick with the Strimzi operator and kafka.<p>Also Confluent is absolutely pricing themselves out of the market. We looked at their self hosted confluent operator and they wanted something like $9k per node, when they do nothing but provide an operator. Insanity.

I&#x27;d like to see a baseline of fio and iperf3 for these same instances so we know how much raw performance is available for disk, network alone and together.<p>Cloud instances have their own performance pathologies, esp in the use of remote disks.<p>As for RP and Kafka performance, I&#x27;d <i>love</i> to see a parameter sweep over both configuration dimensions as well as workload. I know this is a large space, but it needs to be done to characterize the available capacity, latency and bandwidth.

&quot;I hope you come away with a new appreciation that trade-offs exist, there is no free lunch despite the implementation language or algorithms used. Optimizations exist, but you can’t optimize for everything. In distributed systems you won’t find companies or projects that state that they optimized for CAP in the CAP theorem. Equally, we can’t optimize for high throughput, low latency, low cost, high availability and high durability all at the same time. As system builders we have to choose our trade-offs, that single silver-bullet architecture is still out there, we haven’t found it yet.&quot;

Author say &quot;Redpanda incorrectly claim Kafka is unsafe because it doesn’t fsync - it is not true&quot;.<p>If you don&#x27;t Fsync the batch, it&#x27;s possible the server would send response to client saying data was written successfully while the batch is still just in memory and then the server loose power and never write it to disk.<p>Maybe the author have a different definition of unsafe but to me if it&#x27;s not ACID it&#x27;s unsafe!

&quot;...all this is really just benchmarketing, but as I stated before, if no-one actually tests this stuff out and writes about it, people will just start believing it. We need a reality check.&quot;<p>Well said

The biggest point of contention here seems to be over whether kafka can still be considered durable&#x2F;safe when fsync is disabled.<p>Seems like it&#x27;d be valuable to have a trusted third party like <a href="https:&#x2F;&#x2F;jepsen.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;jepsen.io&#x2F;</a> test it out! (not related, just a fan of their work)

A tangent but how do distributed robustness properties in face of communication hiccups compare between Redpanda and Kafka? Eg with Raft apparently you can still fail in presence of asymmetric network failures (like in <a href="https:&#x2F;&#x2F;blog.cloudflare.com&#x2F;a-byzantine-failure-in-the-real-world&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.cloudflare.com&#x2F;a-byzantine-failure-in-the-real-...</a>)

I wonder if there&#x27;s an embedded equivalent for such systems? Something like fasterlog but more mature?

TLDR: &quot;I work at Confluent, the owners of Kafka, and I have determined through my tests that Redpanda&#x27;s performance is greatly exaggerated.&quot;<p>I don&#x27;t think we can get a less reliable or trustworthy set of performance tests than when someone&#x27;s paycheck depends on the outcome of those tests. If Redpanda&#x27;s performance were found to be better, would he really publish the test results?

EDIT: Thank you for clarification. It is a fair 3 node vs 3 node benchmark.<p>Does this benchmark compare both 3 node Kafka against 3 node Redpanda cluster? It&#x27;s unclear.