[nimbius]

Kubernetes is powerful, yes. it is also a feckless rats nest of bolt-ons and ride-alongs. its sharepoint levels of byzantine tuning so complex that, like sharepoint, it comes with its own bespoke administrators that often have little or no knowledge of basic networking or operating systems --only kubernetes--.

- Upgrading a kubernetes cluster may as well be an olympic sport. its so draconian most best practice documentation insists you build a second cluster for AB deployment.

- load balancers come in half a dozen flavours, with the default options bolted at the hip to the cloud cartel. MetalLB is an option, but your admin doesnt understand subnets let alone BGP.

- It is infested with the cult of immutability. pod not working? destroy it. network traffic acting up? destroy the node. container not working? time to destroy it. cluster down? rebuilt the entire thing. At no point does the "devops practitioner" stop to consider why or how a thing of kubernetes has betrayed them. it is assumed you have a football field of fresh bare metal to reinitialize everything onto at a moments notice, failure modes be damned.

what your company likely needs is some implementation of libvirtd or proxmox. run your workloads on rootless podman or (god forbid) deploy to a single VM.

[nazgulsenpai]

I preface this with saying that my experience is all low/medium traffic and single cluster, and I've never had to develop for Kubernetes. But as a sysadmin, I don't mind it at all. I started a new job learned through being given logins to for AWS EKS mobile/web application backend and an on-prem OpenShift for an internal application. The the whole thing mostly works logically once you understand the underlying concepts, and the documentation is pretty good. The only issues I ever had that required external assistance were platform specific quirks (like EKS ALB annotations most recently). Even moved several of our single server workloads over.

[avtar]

> what your company likely needs is some implementation of libvirtd or proxmox. run your workloads on rootless podman or (god forbid) deploy to a single VM.

Even with a single VM, someone's company probably will also want a reverse proxy and certificate management (assuming web services), automated deployments, provide secrets to services, storage volumes, health checks with auto restarts, ability to wire logs and metrics to some type of monitoring system, etc. All of this is possible with scripts and config management tools but now complexity is being managed in different ways. Alternatively use K3s and Flux to end up with a solution that checks all of those boxes while also having the option to use k8s manifests in public clouds.

[bakies]

I dont have any of this experience. I only have to change the version number and the upgrades roll themselves out.

MetalLB is good yes, and admins should have IP knowledge. I ask this in interview questions.

Yes, sheep not pets is the term here. Self healing is wonderful. There's plenty to dig into if you run into the same problem repeatedly. Being able to yank a node out that's misbehaving is very nice from a maintenance pov.

Talos on bare metal to get kubernetes features is pretty good. That's what my homelab is. I hated managing VMs before that.

[otabdeveloper4]

Nix manages to be immutable without restarting everything from scratch.

The complaint isn't immutability, the complaint is that k8s does immutability is a broken, way too granular fashion.

[bakies]

I'm not really clear on the complaint. Is it immutability or not? I'm not saying delete the cluster and start over, I'm saying i can yank a node or destroy a container without (much of) a consequence. Talos is immutable similarly to nix afaik

[devsda]

I guess the complaint is that with resources being immutable, the only standard & recommended way to deal with a problem is to take the resource out.

I know that is the whole point of sheep vs pets but it somehow became the "did you restart the pc" version for operations.

[bakies]

There's only small parts of the typically used parts of the kubernetes api that are immutable and those have good reasons. So I'm still not really sure what issue you're describing.

[dilyevsky]

> MetalLB is an option, but your admin doesnt understand subnets let alone BGP

Maybe get someone competent then? Why are you tasking running onprem setup someone who doesn’t understand basic networking?

[icedchai]

To be fair, BGP is definitely beyond "basic" networking. There is a learning curve.

[dilyevsky]

The subset of it you need to grok to setup metallb is in fact pretty basic

[icedchai]

I actually do agree. I've setup MetalLB in my home lab and it was super simple. I've doing networking since the 90's though, both professionally and as a hobby where I operate my own AS for fun, so I can see how someone else could be intimidated.

[zdragnar]

Odd, we went from a bare instance to VM to a tiny k8s cluster, and k8s was the most stable and easy to administer of the lot.

[themgt]

It is infested with the cult of immutability

Immutability is like violence: if it doesn't solve your problem, you aren't using enough of it.