TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Building a Distributed Log from Scratch: Scaling Message Delivery

130 pointsby tylertreatover 7 years ago

1 comment

majidazimiover 7 years ago
&gt; In Kafka and NATS Streaming, reads (and writes) are only served by the leader. Similarly, Amazon Kinesis supports up to only five reads per second per shard (a shard is Kinesis’ equivalent of a partition). Thus, if we have five consumers reading from the same shard, we’ve already hit our fan-out limit.<p>Long live DistributedLog. One of the great aspects of DistributedLog is the read scaling. Start reading from any replica. Even read proxy node caches records for all consumers. Launch as many read proxies as you need, since read proxies are stateless.