[withinboredom]

I run k8s on bare-metal, and I can say a full disk is certainly possible if you have a service logging a few mb/s. Things will break in fun and interesting ways, data will get irrecoverably corrupted, etc. Your entire cluster will probably even break if said node was the etcd leader. This is pretty easy to reproduce by simply saturating a network and then watching the etcd leader spill its guts in your logs once the network buffers fill up.

> You no longer have dependencies to your VM because you can't have dependencies on your VM OS.

Your containers rely on the OS's kernel and whatever features it was compiled with.

> You will not install normal software on that VM just because you can

If you're paying through the nose for managed k8s, this is true. If not, you'll eventually need to login to a node and diagnose some issue, which means installing things on the node.

> You no longer need to collect all logfiles from VMs because you do it for your services and for your services

Whatever you installed to collect logfiles is getting them from the VM's disk (in /var/log/pods in k3s), unless your container is redirecting them somewhere that isn't stdout.

[hodgesrm]

> If you're paying through the nose for managed k8s, this is true. If not, you'll eventually need to login to a node and diagnose some issue, which means installing things on the node.

Managed Kubernetes on Amazon (EKS) is quite inexpensive: $0.10/hr * 24 hrs/day * 30 days/month = $72/month. Other costs are VMs, networking, and storage, which you would have allocated anyway. There are some downsides like forced upgrades, but cost is not of them for our use cases.

We incidentally don't ever login to Kubernetes nodes using tools like ssh. It's asking for security trouble to have those ports open.