[pelasaco]

I did the same move as you away from k8s to plain proxmox containers and VMs. Professionally i do work with k8s, and see the benefits of it (not always, but i see the use cases), but in my homelab it was consuming a lot of energy. Just dropping the whole k8s, made me save 1 kw energy when idle... I guess mainly because of the active API, shifting workload from different workloads and the whole machinery that happens behind the scenes..

[trallnag]

I would have assumed that observability components (Prometheus, etc) dwarf the control plain when it comes to activity. At least with not extremely dynamic workloads. And you mention that it was due to shifting workloads. If the demands are static, nothing should shift in my experience?

[pelasaco]

Yes, well I am not sure why, but my nodes, kept pushing load from node a, to node b, then back to a. Maybe updates, new certificates, or trying to optimize usage, etc..

[astralkraft]

Proxmox is merely Debian with some fancy curtains. I'm sort of fascinated by this social belief it is something more than that. Although I agree with your sentiment, I'd rather be on XCP-ng or IncusOS as the foundation for containers. I don't believe sacrificing immutability in the host is an improvement over k8s for stability and maintenance.

[pelasaco]

> Proxmox is merely Debian with some fancy curtains. I'm sort of fascinated by this social belief it is something more than that.

You can always reduce anything to "its just linux". But the glue around, the good updates, the stability, the great integration with ceph, make the whole difference..

[catdog]

> Proxmox is merely Debian with some fancy curtains.

It's part of the appeal that it is mostly just Debian under the hood. If I want to run containers I would also not see it as the best choice but it really shines in managing qemu VMs.

[astralkraft]

Proxmox upgrades make K8s appealing, and that's not easily done.

I've been on Incus with my homelab for a few months now, and couldn't be happier. Maybe I should make a post about it, but I don't have much to report. My biggest maintenance outage was forgetting to set WakeOnPower in the BIOS.

Edit: And yes, all I want to do is run containers. Nothing fancy, Nginx Proxy Manager suffices for networking, and Linstor/DRBD handles disk. I wanted to remove hyperscaling complexity, but keep immutability, and so far, life is good.

[catdog]

On low power, mostly idle systems that's definitely a downside. K8s is always busy, probably largely because it's whole architecture is centered around reconciliation loops which contentiously compare the actual state to the desired state. It's not much but enough to prevent the CPU from entering its lowest power states.

[chuckadams]

K8s uses watchers on etcd, not polling, so if there's no changes, there should be no activity. But just collecting metrics like CPU usage is probably a watcher on constantly changing data, like every second or so. So your control nodes are likely to stay warm most of the time, but there's also things like MicroK8s, which is made so you can stop the control plane and the worker containers keep running.