[MPSimmons]

I can expand a little bit, but based on your question, I suspect you may know everything I'm going to type.

In cloud environments, it's pretty common that your cloud provider has specific implementations of Kubernetes objects, either by creating custom resources that you can make use of, or just building opinionated default instances of things like storage classes, load balancers, etc.

It's pretty easy to not think about the implementation details of, say, an object-storage-backed PVC until you need to do it in a K8s instance that doesn't already have your desired storage class. Then you've got to figure out how to map your simple-but-custom $thing from provider-managed to platform-managed. If you're moving into Rancher, for instance, it's relatively batteries-included, but there are definitely considerations you need to make for things like how machines are built from disk storage perspective and where longhorn drives are mapped, for instance.

It's like that for a ton of stuff, and a whole lot of the Kubernetes/OutsideInfra interface is like that. Networking, storage, maybe even certificate management, those all need considerations if you're migrating from cloud to on-prem.

[anang]

I think K8S distributions like K3S make this way simpler. If you’re wanting to run distributed object storage on bare metal the you’re in store for a lot of complexity, with or without k8s.

I’ve ran 3 server k3s instances on bare metal and they work very well with little maintenance. I didn’t do anything special, and while it’s more complex than some ansible scripts and haproxy, I think the breadth of tooling makes it worth it.

[hadlock]

I ran K3S locally during the pandemic and the only issue at the time was getting PV/PVC provisioned cleanly, I think Longhorn was just reaching maturity and five years ago the docs were pretty sparse. But yeah k3s is a dream to work with in 2025 the docs are great and as long as you stay on the happy path and your network is setup it's about as effortless as cluster computing can get.

[anang]

I've been running one for a couple years now, and even in that short of time Longhorn has made huge leaps in maturity. It was/is definitely the weakest link.

Cost wise it's a no brainer. Three servers with 64 GB ECC and 6 cores for the price of three M5 larges. So 192 GB and 18 cores for the price of 24GB and 6 cores.

I think one of reason k8s can get a bad rap is how expensive it is to even approach doing it right with cloud hosting, but to me it seems like a perfect use case for bare metal where there is no built in orchestration.