ArkFlow: High-performance Rust stream processing engine

How do you educate people on stream processing? For pipeline like systems stream processing is essential IMO - backpressure/circuit breakers/etc are critical for resilient systems. Yet I have a hard time building an engineering team that can utilize stream processing; Instead of just falling back on synchronous procedures that are easier to understand (But nearly always slower and more error prone)

Hello, I am the founder of this project and I am very happy that a friend has shared it.<p>ArkFlow is positioned as a lightweight distributed stream processing engine that integrates streaming batches. With the help of datafusion&#x27;s huge ecosystem and ArkFlow&#x27;s scalable capabilities, we hope to build a huge data processing ecosystem to help the community simplify the threshold for data processing, because we always believe that flowing data can generate greater value.<p>Finally, thanks to everyone for their attention.

seems like a simplified equivalent of <a href="https:&#x2F;&#x2F;vector.dev&#x2F;" rel="nofollow">https:&#x2F;&#x2F;vector.dev&#x2F;</a><p>a major difference seems to be converting things to arrow and using SQL instead of using a DSL (vrl)

Previous discussion (46 days ago): <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=43358682">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=43358682</a>

Looks very similar to redpanda-connect&#x2F;benthos

Very similar in appearance to Redpanda Connect (Benthos) which isn’t a bad thing at all. Would be good to elaborate on how error handling is done and what message delivery guarantees it comes with.

&quot;High-performance&quot; is just a meaningless buzzword if you don&#x27;t have any benchmarks or performance comparisons to comparable software

Does this include broker capabilities? If not, what&#x27;s a recommended broker these days (for hosting in the cloud, i.e., an EC2 instance; I know AWS has its own Mqtt Broker but it&#x27;s quite pricy for high volumes).

Looks interesting, how does this compare to arroyo and vector.dev?

So Kafka Connect and Kafka Streams but with rust?

Very interesting - is WARC support on the roadmap?

How do the creators of this plan to make money?