Hydro: Distributed Programming Framework for Rust

There is a nice talk on Youtube explaining the Hydro project (focused mostly on DFIR)<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=YpMKUQKlak0&amp;ab_channel=ACMSIGPLAN" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=YpMKUQKlak0&amp;ab_channel=ACMSI...</a>

This could do with some real-world application examples so I can understand where you might want to apply it.

If there is an intermediary language in the middle with its own runtime, does that mean that we lose everything Rust brings?<p>I thought this would introduce the language to choreograph separate Rust binaries into a consistent and functional distributed system but it looks more like you&#x27;re writing DFIR the whole way through and not just as glue

This is really exciting. Is anyone familiar with this space able to point to prior art? Have people built similar frameworks in other languages?<p>I know different people have worked on dataflow and remember thinking Materialize was very cool and I&#x27;ve used Kafka Streams at work before, and I remember thinking that a framework probably made sense for stitching this all together

Love the effort but I would love an “akka.rs” to eventually make its way into the Rust ecosystem.

How does this compare to timely [0] in terms of data flow? Can you represent control flow like loops in the IR?<p>[0] <a href="https:&#x2F;&#x2F;github.com&#x2F;TimelyDataflow&#x2F;timely-dataflow">https:&#x2F;&#x2F;github.com&#x2F;TimelyDataflow&#x2F;timely-dataflow</a>

So each &quot;process&quot; is deployed as a separate binary, so presumably run as a separate process?<p>If so, this seems somewhat problematic in terms of increased overhead.<p>How is fast communication achieved? Some fast shared memory IPC mechanism?<p>Also, I don&#x27;t see anything about integration with async? For better or worse, the overwhelming majority of code dealing with networking has migrated to async. You won&#x27;t find good non-async libraries for many things that need networking.

looks really cool and I can see a few ways how to use it, especially deploy part which seems unique. Looking forward to more fleshed-out documentation, especially seemingly crucial Streams and Singletons and Optionals part.

I do like the programming model. Do you perform any network optimizations when rewriting applications? Handling network bottlenecks&#x2F;congestion.

How does this compare to using something like Ballista for data pipelines?<p>The latter benefits a lot from building on top of Apache Arrow and Apache Datafusion.

Is this like the BEAM but with rust?

Not sure what problem this is solving. For real applications one would need sth as ray.io for Rust. Academia ppl: let&#x27;s make another data flow framework.