[Nextgrid]

K8S clusters on VMs strike me as odd.

I see the appeal of K8s in dividing raw, stateful hardware to run multiple parallel workloads, but if you're dealing with stateless cloud VMs, why would you need K8S and its overhead when the VM hypervisor already gives you all that functionality?

And if you insist anyway, run a few big VMs rather than many small ones, since K8s overhead is per-node.

[locknitpicker]

> I see the appeal of K8s in dividing raw, stateful hardware to run multiple parallel workloads, but if you're dealing with stateless cloud VMs, why would you need K8S and its overhead when the VM hypervisor already gives you all that functionality?

I think you're not familiar with Kubernetes and what features it provides.

For example, kubernetes supports blue-green deployments and rollbacks, software-defined networks, DNS, node-specific purges and taints, etc. Those are not hypervisor features.

Also, VMs are the primitives of some cloud providers.

It sounds like you heard about how Borg/Kubernetes was used to simplify the task of putting together clusters with COTS hardware and you didn't bothered to learn more about Kubernetes.

[GauntletWizard]

The reason to target k8s on cloud vms is that cloud VMs don't subdivide as easily or as cleanly. Managing them is a pain. K8s is an abstraction layer for that - Rather than building whole machine images for each product, you create lighter weight docker images (how light weight is a point of some contention), and you only have to install your logging, monitoring, and etc once.

Your advice about bigger machines is spot on - K8s biggest problem is how relatively heavyweight the kublet is, with memory requirements of roughly half a gig. On a modern 128g server node that's a reasonable overhead, for small companies running a few workloads on 16g nodes it's a cost of doing business, but if you're running 8 or 4g nodes, it looks pretty grim for your utilization.

[nyrikki]

You can run pods, with podman and avoid the entire k8s stack or even use minikube on a machine if you wanted to. Now that rootless is the default in k8s[0] the workflow is even more convenient and you can even use systemd with isolated users on the VM to provide more modularity and seporation.

It really just depends on if you feel that you get value from the orchestration that full k8s offers.

Note that on k8s or podman, you can get rid of most of the 'cost' of that virtualization for single placement and or long lived pods by simply sharing a emptyDir or volume shared between pod members.

  # Create Pod
  podman pod create --name pgdemo-pod
  # Create client
  podman run -dt -v pgdemo:/mnt --pod pgdemo-pod -e POSTGRES_PASSWORD=password --name client docker.io/ubuntu:25.04
  # Unsafe hack to fix permissions in quick demo and install packages
  podman exec client /bin/bash -c 'chmod 0777 /mnt; apt update ; apt install -y postgresql-client'
  # Create postgres server
  podman run -dt -v pgdemo:/mnt --pod pgdemo-pod -e POSTGRES_PASSWORD=password --name pg docker.io/postgres:bookworm -c unix_socket_directories='/mnt,/var/run/postgresql/'
  # Invoke client using unix socket
  podman exec -it client /bin/bash -c "psql -U postgres -h /mnt"
  # Invoke client using localhost network
  podman exec -it client /bin/bash -c "psql -U postgres -h localhost"

There is enough there for you to test to see that the performance is so close to native sharing unix sockets that way, that there is very little performance cost and a lot of security and workflow benefits to gain.

As podman is daemonless, easily rootless, and on mac even allows you to ssh into the local linux vm with `podman machine ssh` you aren't stuck with the hidden abstractions of docker-desktop which hides that from you it has lots of value.

Plus you can dump a k8s like yaml to use for the above with:

  podman kube generate pgdemo-pod

So you can gain the advantages of k8s without the overhead of the cluster, and there are ways to launch those pods from systemd even from a local user that has zero sudo abilities etc...

I am using it to validate that upstream containers don't have dial home by producing pcap files, and I would also typically run the above with no network on the pgsql host, so it doesn't have internet access.

IMHO the confusion of k8s pods, being the minimal unit of deployment, with the fact that they are just a collection of containers with specific shared namespaces in the general form is missed.

As Redhat gave podman to CNCF in 2024, I have shifted to it, so haven't seen if rancher can do the same.

The point being is that you don't even need the complexity of minikube on VM's, you can use most of the workflow even for the traditional model.

[0] https://kubernetes.io/blog/2025/04/25/userns-enabled-by-defa...

[victorbjorklund]

Because k8s gives you lots of other things out of the box like easy scaling of apps etc. Harder to do on VM:s where you would either have to dedicate one VM per app (might be a waste of resources) or you have to try and deploy and run multiple apps on multiple VM:s etc.

(For the record I’m not a k8s fanatic. Most of the time a regular VM is better. But a VM isn’t = a kubernetes cluster).

[acedTrex]

because if you just do a few huge VMs you still have all the problems that k8s solves out of the box. Except now you have to solve them yourself, which will likely end up being a crappier less robust version of kubernetes.

[reachableceo]

VMs are a standardized system primitive. The “bare metal” bit with RBAC etc through the management layer / hypervisor.

K8s is pallets Vms are shipping containers

Systems / storage / network team can present a standardized set of primitives for any vm to consume that are more or less independent of the underlying bare metal.

Then the VMs can be live migrated when the inevitable hardware maintenance is needed (microcode patching , storage driver upgrades , etc etc etc). With no downtime for the vm itself

[tayo42]

In a large organization their more efficient to run on VMS. You can colocate services that fit together on one machine.

And in reality no one sizes their machines correctly. They always do some handwavey thing like we need 4 cores, but maybe well burst and maybe there will be an outage so lets double it. Now all that utilization can be watched and you can take advantage of over subscription.