Apache Heron: A realtime, distributed, fault-tolerant stream processing engine

If you are interested in hyper-scale event processing but you want to learn it from first principles, I strongly recommend following Martin Thompson&#x27;s talks. Here is an example of one on cluster consensus:<p>&gt; <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=GFfLCGW_5-w" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=GFfLCGW_5-w</a><p>and another on event log architecture:<p>&gt; <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=RlwO6CJbJjQ" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=RlwO6CJbJjQ</a><p>After digging through all of this material and playing around with LMAX Disruptor &amp; Raft, I have been able to develop a really good understanding of how to build these sorts of systems on my own. Fun constraints like &quot;Only one thread actually mutates anything, and its the same one over and over&quot; make for incredibly elegant implementation opportunities. Not having to constantly hunt down exotic thread-safe data structures means that you can focus on building actual value.<p>Latency is the biggest devil you will dance with in this arena, so almost everything you do will be oriented around mitigating that effect. Latency both at the network and inside the CPU&#x2F;memory&#x2F;storage. It applies at every level.

How many distributed stream processing engines is the Apache foundation planning to collect? At this point it seems like there’s more projects that do this (if you squint a bit), than companies with a serious usecase for that type of architecture.

Oh, is this that &quot;next-gen Storm&quot; project that Twitter built? Seems like they&#x27;ve finally given it to Apache, like they did with Storm after buying the company that built it.<p>Based on a quick Wikipedia skim, looks like the answer is &quot;yes&quot;. That explains this bullet point:<p>&gt; Heron is API compatible with Apache Storm and hence no code change is required for migration.

Sometimes, I feel like Apache foundation is like Thanos, collecting all the distributed engines and watch the IT world burn.<p><i>From the beginning, Heron was envisioned as a new kind of stream processing system, built to meet the most demanding of technological requirements, to handle even the most massive of workloads, and to meet the needs of organizations of all sizes and degrees of complexity. Amongst these requirements:<p><pre><code> The ability to process billions of events per minute Extremely low end-to-end latency Predictable behavior regardless of scale and in the face of issue like extreme traffic spikes and pipeline congestion Simple administration, including: The ability to deploy on shared infrastructure Powerful monitoring capabilities Fine-grained configurability Easy debuggability</code></pre> </i><p>I can&#x27;t wait for my next startup interview. We have a requirement of 25 messages per hour, with 10KB per message, you think you can build the ingestion pipeline using Kafka and MongoDB on a 10 node M5d.24xlarge cluster?

I&#x27;ve worked with a bunch of stream processing engines a few years ago (Samza, Kafka Streaming, Spark Streaming, Storm and Flink), and did a comparison between them as part of my internship.<p>IMO, Apache Flink is the most complete project for those use cases. It is well maintained and the devs are very helpful when asked on the mailing lists.

Wonder why this is getting posted today in particular?<p>The quick summary here is that this was a clean-house rewrite of Apache Storm done by an internal team at Twitter. As an open source project history refresher, Apache Storm was originally built by a startup called Backtype, and the project was led by Nathan Marz, the technical founder of Backtype. Then, Backtype was acquired by Twitter, and Storm became a major component for large-scale stream processing (of tweets, tweet analytics, and other things) at Twitter.<p>I wrote a summary of the &quot;interesting bits&quot; of Apache Storm here:<p><a href="https:&#x2F;&#x2F;blog.parse.ly&#x2F;storm&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.parse.ly&#x2F;storm&#x2F;</a><p>However, at a certain point, Nathan Marz left Twitter, and a different group of engineers tried to rethink Storm inside Twitter. There was also a lot of work going on around Apache Mesos at the time. Heron is kind of a merger of their &quot;rethinking&quot; of Storm while also making it possible to manage Storm-like Heron clusters using Mesos.<p>But, I don&#x27;t think Heron really took off. Meanwhile, Storm got very, very stable in the 1.x series, and then had a clean-house rewrite from Clojure to Java in the 2.x series, mainly to improve performance even more. The last stable&#x2F;major Storm release was in 2020.<p>Storm provides a stream processing programming API, a multi-lang wire protocol, and a cluster management approach. But certain cluster computing problems can probably be better solved at the infrastructure layer today. (For example, Storm was developed before the whole container + docker + k8s focus in cloud ops.) That said, it&#x27;s still a very powerful system; on my team, we process 75K+ events per second across hundreds of vCPU cores and thousands of Python processes with sub-second latencies by combining Storm and Kafka with our open source Python project, streamparse.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;Parsely&#x2F;streamparse" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Parsely&#x2F;streamparse</a><p>The core problems Storm solves: modeling data processing as a computation graph; high-speed network communication between threads, processes, and nodes; message delivery guarantees and retry capabilities; tunable parallelism; built-in monitoring and logging; and much more.<p>(Also, I&#x27;d be remiss if I didn&#x27;t mention -- if you&#x27;re interested in stream processing and distributed computing, we are hiring Python Data Engineers to work on a stack involving Storm, Spark, Kafka, Cassandra, etc.) -- <a href="https:&#x2F;&#x2F;www.parse.ly&#x2F;careers&#x2F;python_data_engineer" rel="nofollow">https:&#x2F;&#x2F;www.parse.ly&#x2F;careers&#x2F;python_data_engineer</a>

sometimes I poke fun at the front-end folks and all the new frameworks they&#x27;re constantly chasing.<p>Lately it&#x27;s been starting to feel the same for distributed systems. How many streaming engines are there now under Apache? Four?

I read this as a realtime, distributed, fault-tolerant <i>steam engine</i>, which was hard to imagine.

IMHO, if you need stream processing, start with Apache Flink. It not only offers a much easier to user API compare to Storm and Heron, but also has a superior execution model for time-based, exactly-once, stateful stream processing.

hadoop, kafka, storm, spark, flink, samza, confluent This tastes an aweful lot like javascript framework hell

I find odd so many people complaining about having too many alternatives for streaming systems. If you are not an advanced user of these systems of course you won&#x27;t be able to choose the right one and it won&#x27;t be your job anyways. Think of this, we have dozens of databases out there and there are new ones coming out every week and nobody complains about that. That&#x27;s because there&#x27;s way more people that understands the differences and the use cases so it makes sense.

I would love to know more about how they do stateful stuff, since I haven&#x27;t found anything that can compare to Flink there, but Flink has really poor quality of life stuff compared to some other options (e.g. Flink serialization setup pains vs Beam coders) but their docs kinda trail off here:<p>&gt; Non-idempotent stateful topologies are stateful topologies that do not apply processing logic along the model of &quot;multiply by zero&quot; and thus cannot provide effectively-once semantics. An example of a non-idempotent<p><a href="https:&#x2F;&#x2F;heron.incubator.apache.org&#x2F;docs&#x2F;heron-delivery-semantics#stateful-topologies" rel="nofollow">https:&#x2F;&#x2F;heron.incubator.apache.org&#x2F;docs&#x2F;heron-delivery-seman...</a><p>(I&#x27;m puzzled by their idempotent vs non-idempotent stateful topology description, because if something is mutating an internal state upon receiving events, it will likely be non-idempotent by design... unless they just mean &quot;idempotent stateful&quot; here to refer to keeping track of source&#x2F;output position state and such.)<p>(They also do say that can only support state storage in ZK or local FS, which feels like a likely non-starter compared to Flink for some of my use cases.)

For once almost all commenters so far are thinking the same thing I am. Yet another stream processor from Apache. Might as well create the acronym now. YASP.

I really wonder what people actually use stream processing for, like very concrete examples. My best examples would only go far filtering a stream over a time window to compute an aggregate. My job does not require anything more, it&#x27;s always basic ETL, but I really need to hear specific examples where it&#x27;s useful for others. Been a long time fan of Apache Flink.

So, AWS Pelican coming in 3 months?

Everyone, please note that this is NOT an announcement of Heron entering the Apache Foundation. This happened several years ago, I believe in 2017 (when I was working on it).

Since no one mentioned it in the comments so far, please add Hazelcast Jet to the long list of stream processors : <a href="https:&#x2F;&#x2F;github.com&#x2F;hazelcast&#x2F;hazelcast-jet" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;hazelcast&#x2F;hazelcast-jet</a>

Anyone use this in production except Twitter? Just curious.

Is it another corporate dumping at Apache or does Twitter seem to continue to support it as an open source project there?

Uhh, a triple-take: read this twice as Apache Heroin, and I’m not a druggy.<p>Is it apt? Reader exercise.

For a nonprofit, apache seems to have an unhealthy affection for solving scaling problems that only a few giant rich companies have