Uber’s Real-Time Push Platform

Hey all, one of the authors here. I wanted to provide some more context on the questions asked here.<p>- Most of the tech described in the article is something you will have to do whether you choose gRPC or websockets. Most of the tech was about sharding and maintaining sticky connections and load balancing those connections across servers based on load.<p>- The heartbeat introduced on top of HTTP was needed to detect the broken connection faster and recover quickly as some of the payloads were very latency sensitive. Note that these connections are not 1:1 connections but have multiple hops across low bandwidth mobile networks.<p>- At the time we developed this initially in 2014, most of the WebSockets libraries would fallback to longpolling when the network connections were unstable. We explicitly went away from long polling. Since server-&gt;mobile payloads are the larger of the volumes, we settled in on SSE. Netty and the netty libraries provided most of the implementations of SSE out of the box.

&gt; This first generation of RAMEN server was written in Node.js using Uber’s in-house consistent hashing&#x2F;sharding framework called “Ringpop.”<p>&gt; ...<p>&gt; Additionally, Node.js workers were single threaded and would have elevated levels of event loop lag resulting in a further delay in the convergence of membership information. These issues could result in topology information that is inconsistent and lead to message loss, timeouts, and errors.<p>&gt;<p>&gt; In early 2017, we decided that it was time for a reboot of the server implementation of RAMEN protocol to continue to scale. For this iteration we used the following technologies: Netty, Apache Zookeeper, Apache Helix, Redis and Apache Cassandra.<p>Yet another example of why starting with Java, .NET or similar stacks avoids rewrites.

I don&#x27;t get the point of this article. What they achieved with the RAMEN service can easily be achieved by anyone using gRPC. At the very end of the article they even talk of a version 2 of their service which uses gRPC and seems promising.<p>Realtime push technically exists in every chat app on your mobile phone, so it&#x27;s not a great technical feat either. Am I missing something?

Is it just me or they have overestimated the problem?<p>For Uber’s use case I don’t see a reason to build the system that requires global scale. Users are often residing in the same geolocation. Why build this massive system that works for all instead of building smaller decentralized edge systems that does slower syncs in the back?<p>Very nice read nonetheless.

Ramen v3 prediction:<p>- we separated payload from notification, now client is pulling payload on notification.

as someone that has complained about a lot of uber&#x27;s resume driven development (see comments), I can attest that this system is<p>a.) necessary, and saves the company millions<p>b.) a simplification over the ... unusually ... architected system that came before and did not scale.<p>there&#x27;s a lot of junk technology at many large tech companies, but scaling problems &#x2F;are&#x2F; real when you have hundreds of millions of users and have demanding performance and reliability requirements. RAMEN is a differentiated and necessary part of Uber&#x27;s infrastructure, even if that can&#x27;t be said of a lot of uber eng blog posts.

What would be the second best way to do cover this use case with minimal effort instead of doing it from scratch? For lets say Amsterdam City Sized cab hailing?

I didn’t see them mention why they didn’t go with the more obvious websocket for the actual last mile communication with the device, since they wanted it to be bidirectional.<p>My guess is because doing this in Java is a huge pain whereas it’s practically a breeze in something like elixir.

&gt; We have operated this system with more than 1.5M concurrent connections and pushes over 250,000 messages per second.<p>1.5 million? I thought Uber would have more simultaneous active clients than that.

Instead of switching all the way to gRPC for a return channel they could also have opened another persistent HTTP&#x2F;1.1 connection which can be ready to send whatever is required.<p>gRPC (or more: HTTP&#x2F;2) will win if the amount of request-levle concurrency that is required is extremely high. But in that case also the danger of head-of-line blocking between the various HTTP&#x2F;2 streams gets higher.

&#x27;store network connections in zookeeper&#x27; is actually a compelling value prop for socket infrastructure once all the glue is resolved<p>wonder if this &#x27;fireball&#x27; tool is really a general way to map realtime messages onto schema updates -- if yes, could potentially get uptake on other teams (if their schema format is at all portable)

Uber does seem to reinvent everything for itself - I’d be curious what their ops engineering spend is.<p>I’ve not fully understood how they can lose so much money given their fee per ride is so much higher than most apps would dream of for revenue

The article states: &gt;&quot;These polling APIs ultimately just became a set of payload sharded APIs for the app to poll its features.&quot;<p>I&#x27;m curious would this essentially be sharding on HTTP headers that indicate payload type? Can anyone say?

Genuinely curious coz I don’t understand mobile networks very well — why not just send frames over websockets? You still get a persistent TCP connection and there’s no need for pushing or polling of any kind.

An open-source project (from myself) doing the same with WebSockets: <a href="https:&#x2F;&#x2F;github.com&#x2F;Cretezy&#x2F;dSock" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Cretezy&#x2F;dSock</a>

So much effort to do UDP over HTTP ...<p>No wonder they need 1000s of SWEs.

Why reinvent the wheel instead of using the state of the art RSocket protocol? Plus it handle backpressure for free, being reactive.

Real Time. Over the internet. What.<p>I actually looked it up.<p><a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Real-time_web" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Real-time_web</a><p>It feels like something a SW company wrote up for the sole purpose of making up the idea.<p>The &quot;Difference from real-time computing&quot; is uncited. Hell the entire thing is uncited except for Real Time search, from Google.