ArkFlow – High-performance Rust stream processing engine

I work in one of large tech companies, and I can attest that while the idea seems very neat in theory (esp if your schemas are typed), and even if you define an api for defining new building blocks, sooner or later people realize that they need to dynamically adjust parts of the pipeline, and they write components to dynamically set and resolve these, and then other components on top of these components, and then complements for composing components - and now you forced yourself into implementing a weird and hard to debug functional programming language in yaml which is not a place someone wants to find themselves in :&#x27;(<p>one lesson I learned from this: any bit of logic that defines a computation should prefer explicit imperative code (eg python) over configuration, because you are likely to eventually implement an imperative language in that configuration language anyway

I worked on something very similar for inference on video streams. To avoid the limitations of the config files mentioned in a sibling comment, I added a tool to convert a config to plain Rust. Your primary focus has to be the quality of the Rust API, and the config files are syntactic sugar for the beginning or simpler projects.

Very cool! Seems like a Rust version of something like Bento? [1] Have you done any benchmarking against similar stream processing tools?<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;warpstreamlabs&#x2F;bento" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;warpstreamlabs&#x2F;bento</a>

High Performance: Built on Rust and Tokio async runtime, offering excellent performance and low latency Multiple Data Sources: Support for Kafka, MQTT, HTTP, files, and other input&#x2F;output sources Powerful Processing Capabilities: Built-in SQL queries, JSON processing, Protobuf encoding&#x2F;decoding, batch processing, and other processors Extensible: Modular design, easy to extend with new input, output, and processor components

Reminds me of Pathway <a href="https:&#x2F;&#x2F;pathway.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pathway.com&#x2F;</a>

Good job brother! What do you think you need to implement before it is production-ready?

Rust is increasingly becoming the default language for building infrastructure.

I love the simplicity of this design

Please tell me you are at least aware that tremor exists and that you rebuilt it on purpose?<p><a href="https:&#x2F;&#x2F;www.tremor.rs&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.tremor.rs&#x2F;</a>