I’m one of the long term PMC / committers on mesos.<p>In retrospect I feel this was inevitable to a few key reasons:<p>* k8s was a second system with all the learnings and experience of building such a system at Google for over a decade. Mesos was birthed by grad students and subsequently evolved into its position at Twitter but the engineers driving the project (myself included) did not have experience building cluster management systems. We learned many things along the way that we would do differently a second time around.<p>* Mesos was too “batteries not included”: this fragmented the community, made it unapproachable to new users, and led to a lot of redundant effort. Most users just want to run services, jobs and cron jobs, but this was not part of Mesos and you had to choose from the array of ecosystem schedulers (e.g. Aurora, Marathon, Chronos, Singularity, etc) or building something yourself.<p>* Mesosphere was a VC backed startup and drove the project after Twitter. Being a VC backed startup you need to have a business model that can generate revenue. This led to a lot of tensions and mistrust with users and other vendors. Compare this with Google / k8s, where Google does not need to generate revenue from k8s directly, it can instead invest massive amounts of money and strong engineers on the notion that it will improve Google’s cloud computing business.<p>Even if k8s hadn’t come along, Mesos was ripe to be disrupted by something that was easier to use out of the box, and that had a benevolent home that could unite vendors and other contributors. Mesos could have perhaps evolved in this direction over time but we'll never know now.
It's sad, but expected.<p>This is not about Apache, but a failed open-governance for commercial open-source from Mesosphere. It's not the case with Apache Spark nor Apache Beam, HBase, etc.<p>Mesos (and many other Berkeley AMPLab efforts) had briliant ideas behind it and an elegant implemention that allowed for much more than what Kubernetes was desgined to.<p>Kubernetes was supposed to be scheduler for Mesos and Google invested in it. Let's not forget John Wilkes (Google Borg, Omega, Kubernetes) on the stage at MesosCon in 2014 <a href="https://www.youtube.com/watch?v=VQAAkO5B5Hg" rel="nofollow">https://www.youtube.com/watch?v=VQAAkO5B5Hg</a><p>While I don't know the exact reasons behind Kubernetes becoming the scheduler and resource manager, I think that has very much to do with the stewardship of the Mesos project as well.<p>By 2015/16 most Mesos committers were at Mesosphere. Suffice to say the open-governance was more of pain than a benefit to them. If you were at Apple, Adobe and others that relied on Mesos you didn't have much to say. Everything shifted towards DCOS and everyone was pushed towards commercial licenses.<p>This is said, because sadly Kubernetes didn't fix the whole problem and left us with a distributed system with a text based "API" that requires text patching to manage and borderline clueless about the lifecycle of the services running on top. Yet, it's the best we got :)
I went to see Mesos early in their life in their San Francisco office after a joint customer put us in touch.<p>Never in my life did I meet such an arrogant group of people.<p>First, they left us waiting in reception for an hour.<p>They eventually took the meeting over lunch, where we had to watch them inhaling their free food.<p>Some guy in plastic-leather trousers spent most of the hour lecturing us about all of the multi million deals they were doing, and how they weren't interested in speaking with anyone who couldn't write 7 figure checks.<p>Not once did they ask about our business or even our names if I remember correctly.<p>I had a similar experience with other west coast tech companies. Too arrogant for their own good and not able to put in the hard yards with stodgy old enterprise companies even when they have good technology and are early to market.
I used Mesos for a few years before experiencing Kubernetes. As neat as Mesos was, it was doomed from the start.<p>For one, Kubernetes was, at least to some extent, a rewrite and extraction of functionality built at Google, from their production orchestration ecosystem that is Borg. The fact that Kubernetes was heavily influenced by a successful, large solution in this space allowed it to leapfrog, at least a bit, the competition.<p>Mesos was trying to become a solution seemingly from scratch. I worked with and interacted with a number of Mesos project members and committers, and while they were generally bright folks, their industry experience was surprisingly shallow. In my experience, Mesos changed frequently and there were enough unknown TBD components at the edges of the ecosystem to make it somewhat volatile.<p>Within a year Kubernetes was waaaaay ahead and Mesos started considering pivoting to support k8s.<p>I no longer see a purpose in Mesos, and frankly, that's okay. Too many Apache projects are on eternal life support, and they lower the bar for the rest of the Apache ecosystem, which has sort of begun to earn a poor reputation for quality (ala some of the fringe Boost libraries). Apache Foundation is no longer a label for solid, reliable software.
I preferred Mesos to k8s. I think it's core architecture (a 2-level scheduler) is a better foundation. For the longest time, I felt k8s was effectively an overgrown hobby project that had no place being deployed the way it was. That had me realize something in the shower this morning:<p>k8s is the Rails of the cloud.<p>Back when Rails came out, it too was a bit of a hobby project. When coming from more established enterprise web frameworks, Rails felt like a toy. It didn't have the features, robustness, safety, and scalability of "proper" frameworks.<p>What did Rails do? It was easy to get started and it hid a lot of the boring and painful work of web frameworks at the time.<p>Through the sheer force of will of a massive community, Rails grew up and became something more the toy it started as. I was pretty arrogant in my opinions about the Rails community at first, but then I ended up working on several Rails projects over the years.<p>It still hides a lot under the hood, there are still arguably better technical frameworks out there and plenty of folks use it improperly, when they don't need to, and without really understanding the fundamentals, meaning that they tend to get in trouble when pushing the limits or moving outside the golden path of development.<p>And I feel the same way about k8s. I think it started out without anywhere near the features of similar frameworks. It didn't scale well, was simplistic in it's model, and overly complicated in it's implementation. But it was much more approachable than something like Mesos and answered the question of "why am I containerizing everything?", giving everything a purpose to those who started down the path of Docker. And now it has a huge following and industry behind it.<p>At this point, I've learned that what becomes popular isn't necessarily the "right" or "correct" architecture. There's a lot more to programming trends (and fads) than that. The whole industry seems to want to reinvent the wheel every decade, almost like some sort of planned obsolescence to justify our work. Nevertheless, it's rarely wise to fight against the tide and when you have the enough of the industry moving in a direction, we can make even toys into real tools.
Fun fact: According to the initial Mesos paper: Spark was a demo project to show off Mesos.<p><a href="https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf" rel="nofollow">https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf</a><p>> To validate our hypothesis that specialized frameworks providevalue over general ones, we have also built a new frame-work on top of Mesos called Spark ...
Disclaimer - I used to work for Mesosphere.<p>On the topic of arrogance, I wasn't in that meeting referenced, but having worked with most people there who could have been, and being a Brit who's worked for west coast startups like that for nearly 10 years, it's that American confidence, outlook and drive (along with Sand Hill road) that contributes to so much success out there. I'm a fan, but have to check myself every time i use 'super' as an adjective.<p>Tech vs Product is a good comparison to make. I banged my head many times on DC/OS (the commercial version of Mesos) and underlying Mesos itself. Solid tech, but I'd have liked to have seen the product develop faster, to realise its potential in the enterprise market, which was huge.
Mesos made a ton of new contributions to distributed resource management. Resource offers and application integration can allow high cluster utilization and efficiency. Giving applications the opportunity to take part in resource allocation and scheduling was similar to the Exokernel design, and led to many interesting middleware architectures.<p>Mesos also introduced Dominant Resource Fairness for allocating resources to competing applications, which has become an important concept in CS and specifically computational economics in its own right.<p>Mesos achieved the rare combination of using CS research towards a widely deployed operational system. Its C++ code is simple enough to understand and hack, and seemed like one of those projects that you can/want to reimplement on your own "in a few days".<p>It is sad how quickly it has become obsolete.
The real gem of the Mesos ecosystem was the much lesser known Singularity scheduler from HubSpot.<p><a href="https://github.com/HubSpot/Singularity" rel="nofollow">https://github.com/HubSpot/Singularity</a><p>I've run this at scale, in production since 2015 and it has been absolutely rock solid and does most of the production things you'd want. Unlike commercial products, it was written to run HubSpot's own infrastructure so it does what a production system needs. Really bummed to have to downgrade to something like K8s in the near future.
We used Mesos as our first container orchestration stack. It worked OK but Kubernetes came along and offered a one stop solution for everything. In Mesos you had to use separate projects for container management (Marathon), discoverability (Consul/HAPROXY). It seemed more geared to people that wanted to run their own stacks for such tasks. For a small to medium sized operation it was difficult to solve these issues where really they are issues everyone running a stack of containers has. This was in 2014 so it was early but k8s came along with everything we needed.
I used mesos a while ago across a couple of cheep VPS for personal projects and services and it was great. I especially liked that I could used a shell executor to run services with filesystem access. For many environments this probably didn't make sense but for me I could put some secrets on the filesystem and services could make unix sockets available to nginx for SSL and hostname based forwarding.<p>Also with Nix I could easily install software the specific services needed and it was trivial to integrate into the garbage collector making it much faster than launching a docker container.<p>That being said the project wasn't moving that fast and it was a bit buggly for me and nodes would regularly get into loops where they couldn't authenticate with the master for various reasons (I think there were timing issues and timeouts that caused issues over the internet). Now I'm using Kubernetes but I have all of this complicated virtual networking that I don't need, I'm locked into docker-like containers and the thing is so complicated that I need to pay someone to run it for me.
I wonder what recent large-scale adopters of Mesos are going to do. I know Dropbox deployed Mesos in the last year or so[1], even though it was pretty obvious that the project community was dead. Will the existing users found a new community?<p><a href="https://www.amd.com/system/files/documents/new-epyc-case-study-dropbox.pdf" rel="nofollow">https://www.amd.com/system/files/documents/new-epyc-case-stu...</a>
The framework-based scheduler architecture looked like an interesting concept at first sight. But its advantage (write your custom scheduling policy, along with gaining scalability) evidently wasn't that important to make up for the effort of actually having to write your own framework to run anything. (In this regard its a bit like object storage as a file system replacement narrative, which promises scalability, if you adapt your application).<p>This seems to be true both for the Omega experiments at Google (which I assume still uses Borg), and for Mesos. In the end, Mesos always lacked even a minimal application deployment story. Marathon unfortunateley never got beyond being a shiny UI with a primitive application model and a hacky API.<p>I think if Mesos(-sphere) had recognized that gap in time, and came up with a decent framework for application deployments, instead of telling everyone to write their own framework (in C++, against a hacky code base), Kubernetes might not have had a chance, or at least we would have two alternatives to chose from. Too bad.
Nothing is decided yet. This is a vote thread which _might_ end with someone interested in Mesos stepping in to pick the project up.<p>This hasn't happened in a while with other projects (a whole bunch of Apache projects have just retired in the last few months) but with Mesos there might be a chance.
The vote has been cancelled.<p>New discussion - <a href="https://news.ycombinator.com/item?id=26748191" rel="nofollow">https://news.ycombinator.com/item?id=26748191</a>
Somewhat of an end of an era. Worked at a shop that had one of the largest Mesos fleets in the world. Thousands of physical nodes, petabytes of data. True Mesos in production was something to witness.
Most of the comments here about Mesos are exactly my experience with Kubernetes. Replacing the words and I'm just nodding my head:<p><i>"Kubernetes changed frequently and there were enough unknown TBD components at the edges of the ecosystem to make it somewhat volatile."</i><p><i>"Kubernetes was an interesting project but the defaults were incredibly naive about production environments."</i><p><i>"Kubernetes unfortunateley never got beyond being a shiny UI with a primitive application model and a hacky API."</i><p><i>"A challenge with Kubernetes is that Kubernetes was a piece of technology, a framework at most, instead of a product."</i><p><i>"In Kubernetes you had to use separate projects for container management (kubelet, kube-scheduler, kube-controller-manager, kube-proxy, cri-o), discoverability (etcd). It seemed more geared to people that wanted to run their own stacks for such tasks. For a small to medium sized operation it was difficult to solve these issues where really they are issues everyone running a stack of containers has."</i>