Gazette: Cloud-native millisecond-latency streaming

From reading the docs, this has an IMO surprising design decision: the “journal” is a stream of <i>bytes</i>, where each append (of a byte string) is atomic and occurs in a global order. The bytes are grouped into fragments, and no write spans a fragment boundary.<p>This seems sort of okay if writes are self-delimiting and never corrupt, and synchronization can always be recovered at a fragment boundary.<p>I suppose it’s neat that one can write JSONL and get actual JSONL in the blobs. But this seems quite brittle if multiple writers write to one journal and one malfunctions (aside from possibly failing to write a delimiter, there’s no way to tell who wrote a record, and using only a single writer per journal seems to defeat the purpose). And getting, say, Parquet output doesn’t seem like it will happen in any sensible way.

Gazette is at the core of Estuary Flow (<a href="https:&#x2F;&#x2F;estuary.dev" rel="nofollow">https:&#x2F;&#x2F;estuary.dev</a>), a real-time data platform. Unlike Kafka, Gazette’s architecture is simpler to reason about and operate. It plays well with k8s and is backed by S3 (or any object storage).

I feel a bit paralyzed by Fear Of Missing Io_Uring. There&#x27;s so much awesome streaming stuff about (RisingWave, Materialize, NATS, DataFusion, Velox, neat upstarts like Iggy, many more), but it all feels built on slower legacy system libraries.<p>It&#x27;s not heavily used yet, but Rust has a bunch of fairly high visibility efforts. Situation sort of feels similar with http3, where the problem is figuring out what to pick. <a href="https:&#x2F;&#x2F;github.com&#x2F;tokio-rs&#x2F;tokio-uring">https:&#x2F;&#x2F;github.com&#x2F;tokio-rs&#x2F;tokio-uring</a> <a href="https:&#x2F;&#x2F;github.com&#x2F;bytedance&#x2F;monoio">https:&#x2F;&#x2F;github.com&#x2F;bytedance&#x2F;monoio</a> <a href="https:&#x2F;&#x2F;github.com&#x2F;DataDog&#x2F;glommio">https:&#x2F;&#x2F;github.com&#x2F;DataDog&#x2F;glommio</a><p>Alas libuv (powering Node.js) shipped io_uring but disabled it latter. Seems to have significantly worn out the original author on the topic to boot. <a href="https:&#x2F;&#x2F;github.com&#x2F;libuv&#x2F;libuv&#x2F;pull&#x2F;4421#issuecomment-2225860128">https:&#x2F;&#x2F;github.com&#x2F;libuv&#x2F;libuv&#x2F;pull&#x2F;4421#issuecomment-222586...</a>

More details viewable here: <a href="https:&#x2F;&#x2F;gazette.readthedocs.io&#x2F;en&#x2F;latest&#x2F;" rel="nofollow">https:&#x2F;&#x2F;gazette.readthedocs.io&#x2F;en&#x2F;latest&#x2F;</a>

&gt; the broker is pushing new content to us over a singled long-lived HTTP response<p>Any plans to support websocket?<p><a href="https:&#x2F;&#x2F;gazette.readthedocs.io&#x2F;en&#x2F;latest&#x2F;brokers-tutorial-introduction.html#streaming-reads" rel="nofollow">https:&#x2F;&#x2F;gazette.readthedocs.io&#x2F;en&#x2F;latest&#x2F;brokers-tutorial-in...</a>

What&#x27;s the use case for millisecond-latency streaming? HFT? Remotely driving heavy machinery? Anything else?

Where can I get nanosecond latency streaming?