| I’m one of the long term PMC / committers on mesos. In retrospect I feel this was inevitable to a few key reasons: * k8s was a second system with all the learnings and experience of building such a system at Google for over a decade. Mesos was birthed by grad students and subsequently evolved into its position at Twitter but the engineers driving the project (myself included) did not have experience building cluster management systems. We learned many things along the way that we would do differently a second time around. * Mesos was too “batteries not included”: this fragmented the community, made it unapproachable to new users, and led to a lot of redundant effort. Most users just want to run services, jobs and cron jobs, but this was not part of Mesos and you had to choose from the array of ecosystem schedulers (e.g. Aurora, Marathon, Chronos, Singularity, etc) or building something yourself. * Mesosphere was a VC backed startup and drove the project after Twitter. Being a VC backed startup you need to have a business model that can generate revenue. This led to a lot of tensions and mistrust with users and other vendors. Compare this with Google / k8s, where Google does not need to generate revenue from k8s directly, it can instead invest massive amounts of money and strong engineers on the notion that it will improve Google’s cloud computing business. Even if k8s hadn’t come along, Mesos was ripe to be disrupted by something that was easier to use out of the box, and that had a benevolent home that could unite vendors and other contributors. Mesos could have perhaps evolved in this direction over time but we'll never know now. |
One of the lesson I learnt was that Mesos's two level resource allocations was originally designed for running batch workloads (e.g., spark, mpi, etc.) if you look at the original paper. Use it to run long running services is actually an after thought. We end up finding that we have to do lots of tuning on the first level scheduling algorithm to ensure fairness given that the second level scheduler does not have the full view of the cluster and the first level scheduler does not have enough information to make good decisions. The solution to the problem is actually optimistic offer, which is essentially the k8s model.
Another reason k8s was successful is probably because the golang ecosystem. In mesos, we spent a lot of the energy building basic HTTP layer in C++ due to Mesos's unique threading model. I wish we could have spent those time working on actual useful features.