[yujie1984]

Well said Ben. I am also one of the long term PMC/Committer for the project.

One of the lesson I learnt was that Mesos's two level resource allocations was originally designed for running batch workloads (e.g., spark, mpi, etc.) if you look at the original paper. Use it to run long running services is actually an after thought. We end up finding that we have to do lots of tuning on the first level scheduling algorithm to ensure fairness given that the second level scheduler does not have the full view of the cluster and the first level scheduler does not have enough information to make good decisions. The solution to the problem is actually optimistic offer, which is essentially the k8s model.

Another reason k8s was successful is probably because the golang ecosystem. In mesos, we spent a lot of the energy building basic HTTP layer in C++ due to Mesos's unique threading model. I wish we could have spent those time working on actual useful features.

[bogomipz]

Thanks for the historical perspective. Might you or anyone else be able to recommend any resources that discuss the efforts to tune the two level scheduler for long-running workloads?

You mentioned: >"The solution to the problem is actually optimistic offer, which is essentially the k8s model."

Isn't the K8s model more "choose your QoS model" - BestError, Burstable or Guaranteed? Or am I misunderstanding your comment completely?

I was curious about the this:

>"Another reason k8s was successful is probably because the golang ecosystem. In mesos, we spent a lot of the energy building basic HTTP layer in C++ due to Mesos's unique threading model."

Could you say what was unique about the Mesos threading model?

[yujie1984]

> Isn't the K8s model more "choose your QoS model" - BestError, Burstable or Guaranteed? Or am I misunderstanding your comment completely?

k8s's scheduling model is that scheduler is able to see the entire state of the cluster, thus can optimistically make optimal decisions on scheduling, especially for those long running jobs that are very picky in practice. Although k8s by default only runs the default scheduler, you could in theory run multiple schedulers in parallel (the omega model).

Mesos's pessimistic two level offer model makes it hard for second level scheduler to make optimal decisions because it might not get the right offer it needs. At the same time, first level scheduler lacks application specific information to make the right decision to send the right offer to the second level scheduler, thus the problem. We evaluated many first level scheduling algorithms, and ironically found that "random" first level scheduler sometimes works better than DRF for long running services scheduling.

> Could you say what was unique about the Mesos threading model?

Mesos uses a component called libprocess (think of it as C++ version of erlang). Each actor in the system (mesos master, mesos agent) is single threaded. Thus, all i/o operations need to be non-blocking to not block the actor. This makes it hard to integrate 3rdparty C++ libraries, especially those that involves I/O as they might have a different threading model.

Golang solved this problem using go-routing and bake that into the language. So the golang libraries, especially those involve I/O, are much more composable than C++ IMO.