[kube-system]

That is one of the reasons why. Containers share the kernel and system resources. When you want to start running a bunch of containers in a particular configuration, that's when you'd use a container orchestration tool like kubernetes to define how and where you want those containers to run across multiple systems.

While you could schedule containers manually, or just run your application on VMs or hardware manually, something like kubernetes will let you define rules which it will dynamically evaluate against your infrastructure. You can instruct kubernetes to run your map reduce jobs on different nodes than your user-facing site... and you can give kubernetes an arbitrary number of nodes to work with, and it can scale your workloads for you automatically while also following your rules.

[zinodaur]

I guess I would prefer "kubernetes but with VMs instead of containers". The overhead of running in a VM is not very high, and a hypervisor can restrict resource usage much more effectively - so that we could still bin pack map reduce jobs on the same machines as live site services

[ianburrell]

Kubernetes has pluggable container runtimes. There are ones for running VMs, including the new lightweight VMs. They use standard OCI images.

Using VMs with Kubernetes only makes sense when you need the strict isolation. If you are running own code, then containers are faster. Containers also perform better because they can share resources on host. In Kubernetes, containers can have minimum and maximum limits, which means they can dynamically use space not used by other containers. VMs need to be allocated memory when they start.

[pjmlp]

So does plain Docker runtimes nowadays, specially since the company had to pivot into something else.

[kube-system]

If your kubernetes nodes are VMs then you can do both at the same time for different parts of your application.