Kudu as a More Flexible and Available Kafka-Style Queue

Todd from the Apache Kudu (incubating) team here. I'll check this thread throughout the day in case there are any questions (and try to check the original post for comments as well).

This post is a bit confused about how Kafka replication works. Replication in Kafka is always synchronous in the sense that the cluster internally has a strong notion of which messages are committed and no uncommitted message is handed out to consumers. It is just that the client has the option of writing to Kafka without blocking while the servers commit the message.<p>This is described in more detail here: <a href="http:&#x2F;&#x2F;www.confluent.io&#x2F;blog&#x2F;distributed-consensus-reloaded-apache-zookeeper-and-replication-in-kafka" rel="nofollow">http:&#x2F;&#x2F;www.confluent.io&#x2F;blog&#x2F;distributed-consensus-reloaded-...</a>

Fred, really interesting article! Thanks for posting it.<p>For those in the Bay Area, if you&#x27;d like to hear more about Kudu, I&#x27;m giving a talk about it in Palo Alto tomorrow (Wed) at the Big Data Application Meetup at 6PM: <a href="http:&#x2F;&#x2F;www.meetup.com&#x2F;BigDataApps&#x2F;events&#x2F;227191025&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.meetup.com&#x2F;BigDataApps&#x2F;events&#x2F;227191025&#x2F;</a><p>Also presenting are Julien Le Dem of Dremio on Apache Drill and James Taylor of Salesforce on Apache Phoenix. Should be a really interesting evening!<p>(Don&#x27;t mean to hijack the thread, just thought this might be relevant to some folks. :)<p>Mike

I&#x27;d like to know more about read performance. My understanding is that when you base your system on Raft, you have to choose one of two operating modes:<p>- all reads go through the Raft state machine, which means that reading from one node implies talking to at least a quorum of nodes. Consul is an example of this behavior. It provides strong consistency but reduces the performance gain from horizontal scaling. (for Consul at least it&#x27;s been shown that heavy read loads can cause leadership transitions as the master gets overwhelmed).<p>- Reads are processed by the local node only. This is fast and scalable but is basically &quot;eventual consistency&quot;. A read immediately following a write is not guaranteed to see the write. etcd works this way by default (unless you pass ?quorum=true to your read request)<p>How does Kudu behave (or, rather, how is it <i>intended</i> to behave)?

Would be cool to have some sort of link to the Git repo on the `Kudu` page. Googling &quot;kudu git&quot; gave me <a href="https:&#x2F;&#x2F;github.com&#x2F;cloudera&#x2F;kudu" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;cloudera&#x2F;kudu</a>.

I think the term the title was looking for is &quot;Kafkaesque&quot; :)

The following comment is incorrect.<p>&quot;Unlike Cassandra, Kudu implements the Raft consensus algorithm to ensure full consistency between replicas&quot;<p>Cassandra does offer transactional consistency in the form of PAXOS that achieves the same goal. Additionally PAXOS or RAFT based consistency models aren&#x27;t really necessary if you&#x27;re willing to accept out of order processing and adopt idempotent data models; you only need quorum based consistency.<p>I&#x27;m curious to see what your throughput was in TX&#x2F;s not just bytes&#x2F;s in order to get a more apples to apples comparison.

I see that Kudu is marketed as being for analytics, but it is usable as a general-purpose, primary data store?

FYI, Kudu is also the name of Microsoft Azure&#x27;s app management tool...