Kafka Is Not a Database

Alternatively from Jay Krebs [1] a much more thorough and nuanced discussion that is probably the best send-up on this topic.<p>&quot;So is it crazy to do this? The answer is no, there’s nothing crazy about storing data in Kafka: it works well for this because it was designed to do it. Data in Kafka is persisted to disk, checksummed, and replicated for fault tolerance. Accumulating more stored data doesn’t make it slower. There are Kafka clusters running in production with over a petabyte of stored data.&quot;<p>[1] <a href="https:&#x2F;&#x2F;www.confluent.io&#x2F;blog&#x2F;okay-store-data-apache-kafka&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.confluent.io&#x2F;blog&#x2F;okay-store-data-apache-kafka&#x2F;</a>

Tbh, It&#x27;s a weird blog post coming from the materialize folks, considering they know better.<p>The &quot;event sourced&quot; arch they sketched is missing pieces. Normaly you&#x27;d have single writer instances that are locked to the corresponding kafka partition, which ensure strong transactional guarantees, IF you need them.<p>Throwing shade for maketings sake is something that they should be above.<p>I mean c&#x27;mon, I&#x27;d argue that Postgres enhanced with Materialize isn&#x27;t a database anymore either, but in a good sense!<p>It&#x27;s building material. A hybrid between MQ, DB, backend logic &amp; frontend logic.<p>The reduction in application logic and the increase in reliability you can get from reactive systems is insane.<p>SQL is declarative, reactive Materialize streams are declarative on a whole new level.<p>Once that tech makes it into other parts of computing like the frontend, development will be so much better, less code, less bug, a lot more fun.<p>Imagine that your react component could simply declare all the data it needs from a db, and the system will figure out all the caching and rerendering.<p>So yeah, they have awesome tech with many advantages, so I don&#x27;t get why they bad-mouth other architectures.

If it stores data it&#x27;s a database. Filesystems are databases, MongoDB is a database. LevelDB is a database. Postgres and MySQL are databases. Kafka is a database. They are all very different in features and functionality though.<p>What the authors mean is that kafka is not a traditional database and doesn&#x27;t solve the same problems that traditional databases solve. Which is a useful distinction to make but is not the distinction they make.<p>The reality is that database is now a very general term and for many usecases you can choose to special purpose databases for what you need.

I feel like the inventory thing is a bit of a straw-man because the situation is set out in such a way that you need transactions for it to work. If you find yourself wishing you had a global write-lock on a topic to then of course it won&#x27;t work. Modeling your data for Kafka is work just the same as it is for MySQL. Of course it might not be the best tool for the job but you should at least give it a fair shake.<p>You should be able to post &quot;buy&quot; messages to a topic without fear that it messes up your data integrity. Who cares if two people are fighting over the last item? You have a durable log. Post both &quot;buys&quot; and wait for the &quot;confirm&quot; message from a consumer that&#x27;s reading the log at that point in time, validates, and confirms or rejects the buys. At the point that the buy reaches a consumer there is enough information to know for sure whether it&#x27;s valid or not. Both of the buy events happened and should be recorded whether they can be fulfilled or not.

Ok. I admit using Kafka as DB is not straight forward but just stating it doesn&#x27;t provide ACID functionality is not enough.<p>The example they give is very simplistic. With the correct design of kafka topics and events the problem of the example can be fixed.<p>And according to oracle <a href="https:&#x2F;&#x2F;www.oracle.com&#x2F;database&#x2F;what-is-database&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.oracle.com&#x2F;database&#x2F;what-is-database&#x2F;</a> :<p>&gt; A database is an organized collection of structured information, or data, typically stored electronically in a computer system.<p>So Kafka clearly fits that definition.

I think because software engineers tend to excel at pattern recognition, oftentimes solutions to different problems appear so similar that it seems like with a small amount of abstraction, they can be reused. But it&#x27;s a trap!<p>Everything abstracted to the highest level is the same, but problems aren&#x27;t solved at the highest level.<p>The devil, as they say, is in the details.

One way around this is to make sure your kafka command streams are processed in order, in serial partitioned by an id where you want the concurrency control.<p>Normally you only want concurrency control within certain boundaries.<p>By figuring out the minimum amount transaction and concurrency boundaries you can inch out quite a bit of performance.

This is maybe a silly question, but what&#x27;s the difference between the timely dataflow that Materialize uses and Spark&#x27;s execution engine? From my understanding they&#x27;re doing very similar things - break down a sequence of functions on a stream of data, parallelize them on several machines, and then gather the results.<p>I understand that the feature set of timely dataflow is more flexible than Spark - I just don&#x27;t understand why (I couldn&#x27;t figure it out from the paper, academic papers really go over my head).

Kafka is essentially commit logs, which are at the core of any traditional database engines. Streaming is just turning the gut of DB inside out (mostly for scalability reasons), while DB is wrapped up commit logs that provides higher level functionalities (ACID, Transactions, etc.). It&#x27;s two sides of the same coin, yin and yang of the same thing... But on the practical side of things, yes, if what you needed more are indeed what&#x27;s described in this article, your life would be easier with a traditional DB.

So, the problem really being addressed but not named is that eventing systems give <i>eventual</i> consistency. But sometimes that&#x27;s not good enough. And it&#x27;s OK to admit that and bring in another technology when you need a stronger guarantee than that.<p>The example I was taught with was a booking system, where the inventory management system-of-record was separate from the search system. Search does not need 100% up-to-date inventory. A delay between the last item being booked and it being removed from the search results is acceptable. In fact, it has to be acceptable, because it can happen anyway. If someone books the last item after another hit the search button... There&#x27;s nothing the system can do about that.<p>When actually committing a booking, however, then that must be atomically done within the inventory management system.<p>So, to bring it home, it&#x27;s OK for the search system to be eventually consistent against bookings, and read bookings off of an event stream to update its internal tracking. However, the bookings themselves cannot be eventually consistent without risking a double-booking.

Another potential misuse of Kafka I&#x27;ve been wondering about is how a single Kafka instance&#x2F;cluster is often shared by multiple microservices.<p>On one hand the ability to connect multiple microservices to a central message broker is convenient, but on the the other hand this goes against the microservice philosophy of not sharing subcomponents (databases, etc). I wonder where the lines should be drawn.

Jim Gray disagrees: <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;ftp&#x2F;cs&#x2F;papers&#x2F;0701&#x2F;0701158.pdf" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;ftp&#x2F;cs&#x2F;papers&#x2F;0701&#x2F;0701158.pdf</a>

The architecture of dumping events into Kafka and creating materialized views is a perfect choice for many use cases - e.g. collecting clickstream data and building analytical reports.<p>If ACID is a prerequisite, then lot of things won&#x27;t classify as databases - None of Mongo, Cassandra, ElasticSearch etc. Not even many data-warehouses.

As recently as last year, I worked for a company where the Chief Architect, in his infinite wisdom, had decided that a database was a silly legacy thing. The future looked like Kafka streams, with each service being a function against Kafka streams, and data retention set to infinite.<p>Predictably, this setup ran into an interesting assortment of issues. There were no real transactions, no ensured consistency, and no referential integrity. There was also no authentication or authorization, because a default-configured deployment of Kafka from Confluent happily neglects such trivial details.<p>To say this was a vast mess would be to put it lightly. It was a nightmare to code against once you left the fantasy world of functional programming nirvana and encountered real requirements. It meant pushing a whole series of concerns that isolation addresses into application code... or not addressing them at all. Teams routinely relied on one another&#x27;s internal kafka streams. It was a GDPR nightmare.<p>Kafka Connect was deployed to bridge between Kafka and some real databases. This was its own mess.<p>Kafka, I have learned, is a very powerful tool. And like all shiny new tools, deeply prone to misuse.

Is this really a thing? Do people <i>really</i> try to use Kafka as the system of record for financial transactions or similar data?

This post doesn&#x27;t mention the _actual_ answer, which is to:<p>1) Write a event recording a _desire_ to checkout. 2) Build a view of checkout decisions, which compares requests against inventory levels and produces checkout _results_. This is a stateful stream&#x2F;stream join. 3) Read out the checkout decision to respond to the user, or send them an email, or whatever.<p>CDC is great and all, too, but there are architectures where ^ makes more sense than sticking a database in front.<p>Admittedly working up highly available, stateful stream-stream joins which aren&#x27;t challenging to operate in production is... hard, but getting better.

I want to Upvote this more than once. So much facts into a condensed into a small essay. Good job!<p>Money quote: &quot;Event-sourced architectures like these suffer many such isolation anomalies, which constantly gaslight users with “time travel” behavior that we’re all familiar with.&quot;

This is a bit dumbed down, and ignores the domain terminology required to properly discuss the trade-offs here (which is puzzling given that it links to a post by Aphyr, where you can find incredibly thorough discussions around isolation levels and anomalies).<p>&gt; The fundamental problem with using Kafka as your primary data store is it provides no isolation.<p>This is false. I can only assume the author doesn&#x27;t know about the Kafka transactions feature?<p>To be specific, Kafka&#x27;s transaction machinery offers read-committed isolation, and you get read-uncommitted by default if you don&#x27;t opt-in to use that transaction machinery (the docs: <a href="https:&#x2F;&#x2F;kafka.apache.org&#x2F;0110&#x2F;javadoc&#x2F;index.html?org&#x2F;apache&#x2F;kafka&#x2F;clients&#x2F;consumer&#x2F;KafkaConsumer.html" rel="nofollow">https:&#x2F;&#x2F;kafka.apache.org&#x2F;0110&#x2F;javadoc&#x2F;index.html?org&#x2F;apache&#x2F;...</a>). Depending on your workload, read-committed might be sufficient for correctness, in which case you can absolutely use Kafka as your database.<p>Of course, proving that your application is sound with just read-committed isolation is can be challenging, not to mention testing that your application continues to be sound as new features are added.<p>Because of that, in general I think that the underlying point of this article is probably correct, in that you probably shouldn&#x27;t use Kafka as your database -- but for certain applications &#x2F; use-cases it&#x27;s a completely valid system design choice.<p>More generally this is an area that many applications get wrong by using the wrong isolation levels, because most frameworks encourage incorrect implementations by their unsafe defaults; e.g. see the classic &quot;Feral concurrency control&quot; paper <a href="http:&#x2F;&#x2F;www.bailis.org&#x2F;papers&#x2F;feral-sigmod2015.pdf" rel="nofollow">http:&#x2F;&#x2F;www.bailis.org&#x2F;papers&#x2F;feral-sigmod2015.pdf</a>. So I think the general message of &quot;don&#x27;t use Kafka as your DB unless you know enough about consistency to convince yourself that read-committed isolation is and will always be sufficient for your usecase&quot; would be more appropriate (though it&#x27;s certainly a less snappy title).

&gt; The problem we now have is called write skew. Our reads from the inventory view can be out of date by the time the checkout event is processed. If two users try to buy the same item at nearly the same time, they will both succeed, and we won’t have enough inventory for them both.<p>And you&#x27;ll have exactly the same problem if you&#x27;re using a traditional ACID database: the user saw the item as being available, clicked buy, but it was unavailable by the they went to get it. Using an ACID database doesn&#x27;t gain you anything; you might as well just use Kafka for everything.

Any sufficiently complex software will end up implementing a database.

I had an issue with RabbitMQ where I didn&#x27;t know how my consumer was going to use the data that I was writing to a queue yet (from a producer that was listening on a SocketIO or WebSockets stream), and I was kind of just going to figure it out in an hour or something.<p>Eventually, my buffer ran out of memory and I couldn&#x27;t write anything else to it, and it was dropping lots of messages. I was bummed. Is there a way to avoid this in Kafka?

Elasticsearch is also not a database.

Relevant to the discussion:<p>Martin Kleppmann | Kafka Summit SF 2018 Keynote (Is Kafka a Database?) [1]<p>[1] <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=v2RJQELoM6Y" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=v2RJQELoM6Y</a>

<a href="https:&#x2F;&#x2F;www.postgresql.org&#x2F;docs&#x2F;10&#x2F;rules-materializedviews.html" rel="nofollow">https:&#x2F;&#x2F;www.postgresql.org&#x2F;docs&#x2F;10&#x2F;rules-materializedviews.h...</a>

Kafka is a very nice communication channel. You can dump the results into a database and query it if you need a database.

Well, then you have never heard of ksqlDB. It adds SQL and DB features to Kafka. It is backed by Confluent (LinkedIn) same company that developed Kafka initially.<p><a href="https:&#x2F;&#x2F;ksqldb.io" rel="nofollow">https:&#x2F;&#x2F;ksqldb.io</a>

How does using Kafka come into play when implementing Actor model?<p>Anyone successfully implemented actor model framework over kafka?<p>interested in learning others&#x27; experience

neither has to be.

using the any tool for correct problem requires skills.

I mean, duh? Does Apache Kafka ever made the claim that it is a database?<p>Other things that are not a database: Apache Traffic Server, Apache Mahout, Apache Jakarta, Apache ActiveMQ... hundreds of these exist.