At the computer science level, there's <i>What Every Software Engineer Should Know About Real-Time Data Unifying</i> from 2013:<p><a href="https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying" rel="nofollow">https://engineering.linkedin.com/distributed-systems/log-wha...</a><p>Reading it was an aha. At its core, the log is the simplest thing that might work in a lot of cases.
We use Kafka a lot within the company I work for. I think it's great.
The only thing I miss is fast lookups based on some key and/or the ability for subscribers to only receive messages for certain keys.
This should be labelled as Show HN. One comment - towards the end you say<p>> You might think that Kafka would be using queue data structure internally. It’s not true. Kafka uses a “log” data structure. It is a persistent data structure which allows only appends, no editing, no deletion. In detail, we will cover this some other day.<p>Almost immediately followed by:<p>> Ok, I have a confession . I lied. A Kafka topic is not just a single queue. It’s a combination of queues which helps kafka scale. Every queue is called partition.