[topspin]

For what k8s was designed to do -- herding vast quantities of ephemeral compute resources across a global network -- it's great. That's not my problem with it. My problem is that by being widely misapplied it has stunted the development of good solutions to everything else. K8s users spend their efforts trying to coax k8s to do things it was never intended to do, and so the k8s "ecosystem" has spiraled into this duplicative, esoteric, fragile, and costly bazar of complexity and overengineering.

[rodgerd]

Indeed. One of my rules of thumb is that if it requires permanent storage, k8s is the wrong place, even if it is in-theory possible to do. Dealing with that whole side of things is so error-prone.

[topspin]

I've looked at all the prevailing efforts to wed k8s and block storage and network file systems. It's all nauseating.

On one hand you have network storage vendors (netapp, synology, ceph, whatever) baking up "drivers" aiming at a moving "CSI" target, a spec revised 10 times over seven years as the k8s world muddles its way to retrofitting state to a system that was never intended to deal with state at all. These vendor specific (!) "drivers" are doing laughable things like exec-ing iscsiadm to cobble up storage connections from the host. The only way this could be more "error-prone", as you say, is if Microsoft was selling it.

On the other hand, you have perfectly good, battle tested kernels that have all the necessary vendor independent modules and tools to handle this stuff flawlessly, and no thought given as to how to "orchestrate" them.*

Beyond that we have live migration**, a capability long part of every hypervisor platform (for some apparently unfathomable reason, entirely lost on k8s disciples,) yet completely forgone by container orchestration. Maybe we'll have a working CRIU sometime before I retire... but I'm not holding my breath.

* Kata containers is the closest we've seen to something real here as far as I know, and even there the developers only considered dealing with network attached storage when users started filing issues on it, the main hang-up being that k8s wasn't capable of propagating the necessary volume metadata, until the Kata folks extended k8s to do so.

** Kata containers again: No support for live migration. The staggering irony is that they haven't considered this because k8s has no concept of live migration: there's no extant, standardized way to express live migration in k8s. So a platform (Kata) that proports to facilitate orchestrating VMs foregoes one of the key affordances of VMs because k8s says so (by omission.) That goes directly to my point that the widespread misapplication of k8s is actively retarding other solutions.

Sorry for the rant. Should anyone happen to read all of that, please don't mistake me for an anti-k8s type: I'm not. I'm anti- the foolishness of people trying to bend k8s into something it was never designed to be, to the exclusion of literally everything else in the computing world.