This is a really neat and good article! I did a talk in Sweden the other month about how to build a distributed database, hopefully it may also be fun/useful/informative for others: <a href="https://youtu.be/5fCPRY-9hkc" rel="nofollow">https://youtu.be/5fCPRY-9hkc</a> (it uses CRDTs instead, so is a counter-part that isn't totally ordered and isn't append-only).<p>I like the OP article though, cause I learned about NATS streaming, which I hadn't heard of before - just Kafka. Will have to check it out.
I have to admit that I only recently got familiar with logs. I was designing a B+Tree[1] in Python for fun and was struggling to make it survive crashes: a single insertion in a tree often results in multiple page writes which is not atomic.<p>The solution to this problem is simple and elegant with a Write-Ahead Log. Every page write is appended to the log and only merged back into the tree file when it's sure that the log is safely written to storage.<p>SQLite has an extensive documentation of its WAL file format, which is great for learning.<p>[1] <a href="https://github.com/NicolasLM/bplustree" rel="nofollow">https://github.com/NicolasLM/bplustree</a><p>[2] <a href="https://www.sqlite.org/fileformat.html#the_write_ahead_log" rel="nofollow">https://www.sqlite.org/fileformat.html#the_write_ahead_log</a>
This is a good introduction, and a very useful abstraction. At Morgan Stanley we have built a PL/compiler and tools around a method of logging like this -- logging algebraic data types and live querying out of them with Haskell-like comprehensions/pattern-matching/etc:<p><a href="https://github.com/Morgan-Stanley/hobbes" rel="nofollow">https://github.com/Morgan-Stanley/hobbes</a>
One question about the index file, "in Kafka, the index uses 4 bytes for storing an offset relative to the base offset and 4 bytes for storing the log position." Isn't the relative offset to the base offset already pointing to the physical location of the message in the segment file? What's the purpose of the second 4-byte field log position?