Heroku Kafka

As a former user of Kafka, this is awesome and it would have been a huge help for our company if this was available then. I&#x27;m glad to hear that a company is offering Kafka as opposed to other propriety versions(AWS Kinesis etc).<p>One thing is odd though, there is no mention of disk space at all and only a configuration of retention time. One of Kafka&#x27;s best features is the use of disk to store large amounts of messages, you are not RAM bound. Heroku seems to only allows you to set retention times? This could be awesome if they are giving you &quot;unlimited&quot; disk space, but could also be a beta oversight. Interested to see how this progresses.

&gt; <i>What is Kafka?</i><p>&gt; Apache Kafka is a distributed commit log for fast, fault-tolerant communication between producers and consumers using message based topics. Kafka provides the messaging backbone for building a new generation of distributed applications capable of handling billions of events and millions of transactions<p>Can anyone translate this into meaningful English for me?

I love Heroku and everything they are doing, it&#x27;s doubtless a push forward for the web as a whole. However, the pricing for hobby sites (including SSL) is crazy from a personal point of view so I&#x27;m slowly moving my projects out of it [1][2]. I wish they had some kind of &quot;Hobby Bundle&quot;.<p>[1] <a href="http:&#x2F;&#x2F;umbrellajs.com&#x2F;" rel="nofollow">http:&#x2F;&#x2F;umbrellajs.com&#x2F;</a> [2] <a href="http:&#x2F;&#x2F;picnicss.com&#x2F;" rel="nofollow">http:&#x2F;&#x2F;picnicss.com&#x2F;</a>

Kudos to Heroku. As someone who has had to make Kafka into a managed service, I know what a pain it is (I&#x27;m not a Kafka fan for a lot of reasons) to administer in a cloud environment.

For anyone wanting to play with Kafka, Spotify&#x27;s Kafka container was an invaluable resource for getting me up and running with Kafka. All the Zookeeper dependencies are taken care of allowing you to just start playing with Kafka right away. <a href="https:&#x2F;&#x2F;github.com&#x2F;spotify&#x2F;docker-kafka" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;spotify&#x2F;docker-kafka</a> <a href="https:&#x2F;&#x2F;hub.docker.com&#x2F;r&#x2F;spotify&#x2F;kafka&#x2F;" rel="nofollow">https:&#x2F;&#x2F;hub.docker.com&#x2F;r&#x2F;spotify&#x2F;kafka&#x2F;</a>

This will be interesting to try out. I&#x27;ve used all the major cloud event&#x2F;logging systems (Kinesis, Azure EventHubs, etc) and so far Google PubSub is the best in features and performance.<p>Only downside with Google Pubsub can be latency (which I&#x27;m working on fixing by building a gRPC driver) but Kafka has proven to be too complicated to maintain in-house. If heroku can provide the speed without the ops overhead, it&#x27;ll be some good competition to Google&#x27;s option.<p>Also want to note that Jay Kreps who helped build Kafka at LinkedIn is now behind <a href="http:&#x2F;&#x2F;www.confluent.io&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.confluent.io&#x2F;</a> which is like a better&#x2F;enterprise version of Kafka.

For those wondering (all imo and only best guess)<p>The biggest advantage of kafka is that all of the heroku marketplace all of a sudden becomes &quot;plug and play&quot;<p>Essentially it&#x27;s the &quot;backend data&quot; equivalent of what segment does for &quot;frontend data&quot;.<p>Example: What&#x27;s the benefit of having a graphDB service in the marketplace if most people dont want to &#x2F; cant invest engineering in keeping the data in (realtime) sync.<p>With kafka they can establish standards that all partners can adapt to, they will simply offer piping of all heroku postgres&#x2F;redis changes.

Does anyone know if Kafka has improved on their data loss issues since tested by Aphyr? <a href="https:&#x2F;&#x2F;aphyr.com&#x2F;posts&#x2F;293-jepsen-kafka" rel="nofollow">https:&#x2F;&#x2F;aphyr.com&#x2F;posts&#x2F;293-jepsen-kafka</a><p>A quote from the article: &quot;At the end of the run, Kafka typically acknowledges 98–100% of writes. However, half of those writes (all those made during the partition) are lost.&quot;

I&#x27;ve wondered why there isn&#x27;t a &quot;big player&quot; in the cloud space for this. Felt like a hole.<p>My operating theory is that the people who would really make use of something like this have grown beyond managed offerings and would take it in house. For smaller operations Redis is more than enough for pub&#x2F;sub. Ditto for SQS for externally triggered eventing.

Is the pricing public?

Can somebody provide a real-life use case for Kafka? I&#x27;ve seen comparisons between Redis, but what specifically does Kafka solve that Redis cannot?

The comments in this thread are funny;<p>Hey, what is Kafka?<p>&quot;It&#x27;s a distributed logging system, not a message queue&quot;<p>Ok, what&#x27;s the use case?<p><i>describes a case when its used as a message queue</i>

Kafka vs Redis. I&#x27;ve only used Redis... what should I know?

What kind of companies or startups usually use this service?

All kafkaesque to me ;-&gt;

My impression of Kafka was that this thing is bloated. How it compares to something like NSQ?