[esseph]

Try it on bare metal where you're managing the distributed storage and the hardware and the network and the upgrades too :)

[JohnMakin]

Why would you want to do that though?

On cloud, in my experience, you are mostly paying for compute with managed kubernetes instances. The overhead and price is almost never kubernetes itself, but the compute and storage you are provisioning, which, thanks to the control plane, you have complete control over. what am i missing?

I wouldn’t dare try to with a small shop try to self host a production kubernetes solution unless i was under duress. But I just dont see what the control plane has to do with it. It’s the feature that makes kubernetes worth it.

[LelouBil]

I am in the process of redoing all of my self-hosting (cloud storage, sso, media server, and a lot more), which previously was a bunch of docker compose files deployed by Ansible. This quickly became unmanageable.

Now I almost finished the setting up part using a single-node (for now) Kubernetes cluster running with Talos Linux, and all of the manifest files managed with Cue lang (seriously, I would have abandoned it if I had not discovered Cue to generate and type check all of the yaml).

I think Kubernetes is the right solution for the complexity of what I'm running, but even though it was a hassle to manage the storage, the backups, the auth, the networking and so on, I much prefer having all of this hosted at my house.

But I agree with the control plane part, just pointing out my use case for self-hosting k8s

[esseph]

I was responding to:

> even managing a few hundred nodes and maybe 10,000 containers, relatively small - I update once a year and the managed cluster updates machine images and versions automatically. Are people trying to self host kubernetes for production cases, and that’s where this pain comes from?

Much of the pain of kubernetes is not the day to day care and feeding of the applications most of the time, it's managing the cluster itself, upgrades, hardware, etc.

Many regulated industries can not run certain workloads in "the cloud", hence where the pain of running kubernetes (at least at first) comes from.

[esseph]

May need low latency, may have several large data warehouses different workloads need to connect to, etc.

Regulated industries, transportation and logistics companies, critical industries, etc.

[lillecarl]

Tinkerbell / MetalKube, ClusterAPI, Rook, Cilium?

A control plane makes controlling machines easier, that's the point of a control plane.

[esseph]

I'm not debating the control plane, I'm responding to the question about "are people running it themselves in production?” (non-managed on-prem infra)

It can be challenging. Lots and lots of knobs.