Using logs to build a solid data infrastructure

I&#x27;ve been really interested in this architecture since Jay Kreps&#x27; blog post on it. One part that I&#x27;m less clear on is how this fits in with request-response style communication between, say, a Web browser and a Web server.<p>In a simple Web-app-writes-to-DB scenario, it&#x27;s easy to read my writes, but with a async log processing system, how am I supposed to organize my code so I can read my writes and respond with useful information?<p>Maybe the solution is to eschew request-response entirely and have all requests return 200, then poll or use two-way communication?<p>Alternatively, I could have my log-appending operation return a value indicating the position in the totally-ordered log, which I could pass to the query interfaces as a way of indicating &quot;don&#x27;t return until you&#x27;ve processed at least to here.&quot; Does anyone do that?<p>Am I totally off base here? I&#x27;d love to hear from anyone who is using these kinds of systems today.

I&#x27;ve really learned a lot from these Confluent posts about building log-based architectures, but I feel like they&#x27;re rehashing the same high-level architectural ideas again and again.<p>I&#x27;m already sold on this idea, and would really love to see more posts that get into the nitty-gritty details of how to integrate Kafka, how to migrate an existing infrastructure, case studies, sample code, etc. It all seems very handwavy otherwise.

Has anyone thought of how the distributed commit log can be extended to client-side FRP?<p>Elm has this notion of signal – <a href="http:&#x2F;&#x2F;elm-lang.org&#x2F;learn&#x2F;What-is-FRP.elm" rel="nofollow">http:&#x2F;&#x2F;elm-lang.org&#x2F;learn&#x2F;What-is-FRP.elm</a> – which is really a stream of changing values, that are used to construct varying model and its rendered view (virtual DOM and all that).<p>I am wondering if we can merge these two notions – signals and commit logs. Consequently, this would replace the traditional &quot;request-response&quot; model in REST API with nothing but signals, thus leveraging the simplicity of FRP for the whole application. Client side Elm code does a &quot;send&quot; on the signal that is connected to the server-side commit log, and also reads from another signal connected to another log that receives new data (added from various places).

Martin&#x27;s talks and blog posts are aways awesome. I&#x27;m really excited to see how this plays out for real applications.<p>The one thing I&#x27;m always somewhat confused by though is how a &quot;totally ordered log&quot; intersects with the reality of a partitioned log. The simplicity of a log seems to break down a bit when you partition.<p>For instance, imagine I want to implement multi-key transactions on top of a distributed datastore. With a totally ordered log this is easy. But with a partitioned log, it becomes much harder.<p>Alternatively, imagine I want to implement a collaborative editing app like Google Docks or something like Slack. A natural design would be to to have millions of independent logs. I can then replay logs to get current state and watch logs to keep it updated. But as far as I&#x27;m aware, partitioned logs like Kafka do not actually support millions of topics. So there&#x27;s no way to replay a log for something like a channel or document.

<p><pre><code> if you want to build a new derived datastore, you can just start a new consumer at the beginning of the log, and churn through the history of the log, applying all the writes to your datastore. </code></pre> For high-throughput environments with lots of appends to the log, how do you get around the ever-increasing size of your log file? I know the traditional answer is to take a periodic snapshot and compact the previous data, but is that built in to tools like Kafka?

The proposed architecture really works well for me. I&#x27;ve used it for a couple of projects now.<p>To throw around some terms for those interested in reading up&#x2F;background: - The separation of writes (through log) from reads (through any of the consumers) is sometimes called: CQRS (command query responsibility separation) - having a centralized log as the defining store for updates&#x2F; change events is sometimes called: eventsourcing - as mentioned in article: elastic search as a consumer of the log, which only gets updates through the log, is an example of an Eager Read Derivation.<p>All defined on site of Fowler.<p>Glad to see this getting more attention. Asked about usage for Kafka as an eventsource here some time ago. Includes insightful answer of Kafka author. <a href="http:&#x2F;&#x2F;stackoverflow.com&#x2F;questions&#x2F;17708489&#x2F;using-kafka-as-a-cqrs-eventstore-good-idea" rel="nofollow">http:&#x2F;&#x2F;stackoverflow.com&#x2F;questions&#x2F;17708489&#x2F;using-kafka-as-a...</a>

That talk was great. We introduced Kafka at my work probably 3-4 months ago, at first only to track events from our webservices, but eventually it became the backbone of communication between our services.<p>The Java library for the consumer part, still based on the Scala code, is not that great though. They&#x27;re rewriting a Java-only library, which is much nicer to use, but I&#x27;m not sure when it&#x27;ll be stable.

I have actually found myself thinking a lot about logs lately and how I can end up using them for a lot of problems, and are sometimes very simple to implement. But I always wondered if I was actually using the right tool for the job...<p>I had no idea they had such far-reaching implications and so many jobs where this <i>is</i> the right tool.<p>Neat!

Isn&#x27;t this just what datomic automatically does?

What is used to create those images?<p>They look like whiteboard pictures but not quite.

Could someone compare Apache Kafka to Eris Industry&#x27;s centralized blockchain model?