How do you educate people on stream processing? For pipeline like systems stream processing is essential IMO - backpressure/circuit breakers/etc are critical for resilient systems. Yet I have a hard time building an engineering team that can utilize stream processing; Instead of just falling back on synchronous procedures that are easier to understand (But nearly always slower and more error prone)
Hello, I am the founder of this project and I am very happy that a friend has shared it.<p>ArkFlow is positioned as a lightweight distributed stream processing engine that integrates streaming batches. With the help of datafusion's huge ecosystem and ArkFlow's scalable capabilities, we hope to build a huge data processing ecosystem to help the community simplify the threshold for data processing, because we always believe that flowing data can generate greater value.<p>Finally, thanks to everyone for their attention.
seems like a simplified equivalent of <a href="https://vector.dev/" rel="nofollow">https://vector.dev/</a><p>a major difference seems to be converting things to arrow and using SQL instead of using a DSL (vrl)
Very similar in appearance to Redpanda Connect (Benthos) which isn’t a bad thing at all. Would be good to elaborate on how error handling is done and what message delivery guarantees it comes with.
Does this include broker capabilities? If not, what's a recommended broker these days (for hosting in the cloud, i.e., an EC2 instance; I know AWS has its own Mqtt Broker but it's quite pricy for high volumes).