[drewrobb]

I'm surprised that the scaling story of k8s/(+etcd?) is still so far behind mesos/zk. There have been mesos clusters at over 10k Nodes for several years now.

I have never personally needed more than a few hundred mesos agents, but these have been added without any noticeable impact on our extremely modestly provisioned (and multi purpose) zk cluster or any other components.

Has anyone used both systems and can speak to any advantages of k8s for these types of workloads?

Also is anyone using some kind of torrent approach as a more reasonable solution to avoid network bottlenecks when distributing big docker images to a large number of nodes?

[jo909]

A lot of the issues were kind of "external" and while worth thinking about for every deployment, not really something the k8s project can do much about other than warn in the documentation.

  - disk latency
  - monitoring queries
  - homemade autoscaler killing all etcd nodes
  - custom scheduling policy moving many kubedns processes to the same node
  - unusually large docker images
  - "sharing" gcr.io request quotas because of Azure NAT IPs

That's not to say that Mesos is not indeed scaling better or easier. I don't know enough about Mesos.