There is a nice talk on Youtube explaining the Hydro project (focused mostly on DFIR)<p><a href="https://www.youtube.com/watch?v=YpMKUQKlak0&ab_channel=ACMSIGPLAN" rel="nofollow">https://www.youtube.com/watch?v=YpMKUQKlak0&ab_channel=ACMSI...</a>
If there is an intermediary language in the middle with its own runtime, does that mean that we lose everything Rust brings?<p>I thought this would introduce the language to choreograph separate Rust binaries into a consistent and functional distributed system but it looks more like you're writing DFIR the whole way through and not just as glue
This is really exciting. Is anyone familiar with this space able to point to prior art? Have people built similar frameworks in other languages?<p>I know different people have worked on dataflow and remember thinking Materialize was very cool and I've used Kafka Streams at work before, and I remember thinking that a framework probably made sense for stitching this all together
How does this compare to timely [0] in terms of data flow? Can you represent control flow like loops in the IR?<p>[0] <a href="https://github.com/TimelyDataflow/timely-dataflow">https://github.com/TimelyDataflow/timely-dataflow</a>
So each "process" is deployed as a separate binary, so presumably run as a separate process?<p>If so, this seems somewhat problematic in terms of increased overhead.<p>How is fast communication achieved?
Some fast shared memory IPC mechanism?<p>Also, I don't see anything about integration with async?
For better or worse, the overwhelming majority of code dealing with networking has migrated to async. You won't find good non-async libraries for many things that need networking.
looks really cool and I can see a few ways how to use it, especially deploy part which seems unique. Looking forward to more fleshed-out documentation, especially seemingly crucial Streams and Singletons and Optionals part.
How does this compare to using something like Ballista for data pipelines?<p>The latter benefits a lot from building on top of Apache Arrow and Apache Datafusion.
Not sure what problem this is solving. For real applications one would need sth as ray.io for Rust. Academia ppl: let's make another data flow framework.