Damn that's some scope creep if I ever saw it: 'try sending Arrow frames end to end' => 'rewrite the otel pipeline in rust'. Seems like the goals of the contributors don't exactly align with the goals of the project.<p>Kind of a bummer - one thing i was hoping to come out of this was better Arrow ecosystem support for golang.
I've just started exploring adding OpenTelemetry support to the Comet subproject of DataFusion. I'm excited to see the integration with Apache Arrow (Rust) and potentially DataFusion in the future.
Thanks for sharing this — it’s a really promising direction. The advantages of Arrow for OTLP, especially when used end-to-end, are compelling given the protocol overhead of OTLP.<p>We’ve been thinking along similar lines with the use of Rust, particularly for OpenTelemetry collection in environments where high performance and low resource overhead are critical, such as edge and serverless. With that in mind, we’ve open-sourced a lightweight OpenTelemetry collector written in Rust to address these use cases. We’ve also developed a native Lambda extension around it, and have seen encouraging interest from folks aiming to improve cold start times.<p>The project is still fairly early, but we’re optimistic that Rust can open up new opportunities for efficient observability pipelines. Vendors like Datadog are also moving in this direction with their Lambda extension and appear to be adopting Rust more broadly for data-plane components.<p>If this resonates, feel free to take a look here: <a href="https://github.com/streamfold/rotel">https://github.com/streamfold/rotel</a>. We’d love to hear your thoughts on how this could be useful.
Integrations with OTLP are critical to driving adoption and probably one of the biggest pain points we've encountered when adopting it ourselves (and encouraging others to the same).<p>Adopting OTLP without third-party support is pretty time consuming, especially is your tech stack is large and/or varied.<p>Re runtimes: curious about this too. Feels like the right direction if you’re optimizing a telemetry pipeline.
> We are interested in making OTAP pipelines safely embeddable, through strict controls on memory and through support for thread-per-core runtimes.<p>I'm curious about the thread-per-core runtimes, are there even any mature thread-per-core runtimes in Rust around?
Man Ive dipped my toes into this too, and yeah, the way everyone wants different things always shakes things up fast. Kinda love seeing where it all ends up tbh.