Open Sourcing Peloton, Uber’s Unified Resource Scheduler

Is it just me or is every big release from Uber just a custom rewrite of an existing technology? It seems their engineering department has a large not-invented-here attitude. I could be wrong - they're certainly large enough to have custom requirements that aren't met with what's on the market but the pattern is just becoming suspect.

I’ve worked with Mesos pretty extensively before and when Uber first announced Peloton last year I was intrigued. Peloton seems to be a wrapper around Mesos that allows for running smaller, unique jobs without having to write a Mesos framework for each. Writing a Mesos framework for every small job you have can get annoying when you just want to define how your job should run and don’t really care for the resources or task allocation of the job, and it seems like Peloton solves this on Mesos. It’s similar to YARN but not limited to Hadoop. It would have been useful for the project that I worked on because it was more geared for our use case and shifting from Mesos to k8s would’ve been a huge engineering project.

Just call it (Uber) customized mesos. I find this article somehow deceiving and boring. I am pretty sure I can run this peloton thingy with most Mesos API calls.

What exactly is Uber running in its clusters?<p>- Route Optimization<p>- Demand Forecasting<p>- Rider Hotspot Prediction<p>This post doesn&#x27;t exactly tell us the true nature of their workloads (other than the crude categorization - batch, stateful, stateless), nor does it talk about the inflection points where off-the-shelf solutions don&#x27;t cut it anymore and such customization is required. I mean some <i>before &amp; after numbers &#x2F; graphs</i> on resource utilization would have really helped.

At first glance I thought the &quot;resources&quot; here might be Uber drivers, which would be interesting.<p>Is this thing&#x2F;Apache Mesos abstract enough to allow for such a use?

This sounds a bit like Two Sigma’s Cook scheduler for Mesos. <a href="https:&#x2F;&#x2F;github.com&#x2F;twosigma&#x2F;Cook" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;twosigma&#x2F;Cook</a>

Immediate reaction: Well, now there&#x27;s Peloton, the fitness tech company, Peloton, the self-driving truck caravan company, and Peloton, the cluster scheduler...

My first reaction was &quot;this just sounds like Mesos?&quot;. And it&#x27;s cited in their page (which on first read I thought meant they were trying to act as a single pane of glass for Mesos&#x2F;k8s&#x2F;etc.):<p><a href="https:&#x2F;&#x2F;eng.uber.com&#x2F;peloton&#x2F;" rel="nofollow">https:&#x2F;&#x2F;eng.uber.com&#x2F;peloton&#x2F;</a><p>In the OP blog post though, they assert &quot;to our knowledge, there is no other open source scheduler which combines all types of workloads for web-scale companies like Uber.&quot;<p>And then, when you dig...it&#x27;s just Mesos. They built a framework for Mesos. So, that&#x27;s cool. But <i>man</i>, the puff piecery borders on dishonesty. I mean--Singularity has existed, and is implemented at very large scales, for a while. I&#x27;m sure Peloton is a fine scheduler, but there&#x27;s a lot of huffing-one&#x27;s-own-farts in the documentation here.

You mean not a bike company?!

Peloton should make a framework called Uber in response.

Would people use open source stuff from a morally questionable company? Especially when its just a re-write of existing technology posted to a different github repo?<p><i>EDIT</i> People get so up in arms about Google and Microsoft working with China and the military, but Uber has done some horrendous stuff on their own. Just curious where people think the line is OK to be.