[topspin]

Isn't this discussion based on a false dichotomy? I, too, use VMs to isolate customers, and I use containers within those VMs, either with or without k8s. These tools solve different problems. Containers solve software management, whereas VMs provide a high degree of isolation.

Container orchestration is where I see the great mistake in all of this. I consider everything running in a k8s cluster to be one "blast domain." Containers can be escaped. Faulty containers impact everyone relying on a cluster. Container orchestration is the thing I believe is "overused." It was designed to solve "hyper" scale problems, and it's being misused in far more modest use cases where VMs should prevail. I believe the existence of container orchestration and its misapplication has retarded the development of good VM tools: I dream of tools that create, deploy and manage entire VMs with the same ease as Docker, and that these tools have not matured and gained popularity because container orchestration is so easily misapplied.

Strongly disagree about containers and dev/deployment ("NOT check"). I can no longer imagine development without containers: it would be intolerable. Container repos are a godsend for deployment.

[rodgerd]

> Container orchestration is the thing I believe is "overused." It was designed to solve "hyper" scale problems, and it's being misused in far more modest use cases where VMs should prevail.

As a relatively early corporate adopter of k8s, this is absolutely correct. There are problems where k8s is actually easier than building the equivalent capability elsewhere, but a lot of uses it's put to seem to be driven more by a desire to have kubernetes on one's resume.

[topspin]

For what k8s was designed to do -- herding vast quantities of ephemeral compute resources across a global network -- it's great. That's not my problem with it. My problem is that by being widely misapplied it has stunted the development of good solutions to everything else. K8s users spend their efforts trying to coax k8s to do things it was never intended to do, and so the k8s "ecosystem" has spiraled into this duplicative, esoteric, fragile, and costly bazar of complexity and overengineering.

[rodgerd]

Indeed. One of my rules of thumb is that if it requires permanent storage, k8s is the wrong place, even if it is in-theory possible to do. Dealing with that whole side of things is so error-prone.

[topspin]

I've looked at all the prevailing efforts to wed k8s and block storage and network file systems. It's all nauseating.

On one hand you have network storage vendors (netapp, synology, ceph, whatever) baking up "drivers" aiming at a moving "CSI" target, a spec revised 10 times over seven years as the k8s world muddles its way to retrofitting state to a system that was never intended to deal with state at all. These vendor specific (!) "drivers" are doing laughable things like exec-ing iscsiadm to cobble up storage connections from the host. The only way this could be more "error-prone", as you say, is if Microsoft was selling it.

On the other hand, you have perfectly good, battle tested kernels that have all the necessary vendor independent modules and tools to handle this stuff flawlessly, and no thought given as to how to "orchestrate" them.*

Beyond that we have live migration**, a capability long part of every hypervisor platform (for some apparently unfathomable reason, entirely lost on k8s disciples,) yet completely forgone by container orchestration. Maybe we'll have a working CRIU sometime before I retire... but I'm not holding my breath.

* Kata containers is the closest we've seen to something real here as far as I know, and even there the developers only considered dealing with network attached storage when users started filing issues on it, the main hang-up being that k8s wasn't capable of propagating the necessary volume metadata, until the Kata folks extended k8s to do so.

** Kata containers again: No support for live migration. The staggering irony is that they haven't considered this because k8s has no concept of live migration: there's no extant, standardized way to express live migration in k8s. So a platform (Kata) that proports to facilitate orchestrating VMs foregoes one of the key affordances of VMs because k8s says so (by omission.) That goes directly to my point that the widespread misapplication of k8s is actively retarding other solutions.

Sorry for the rant. Should anyone happen to read all of that, please don't mistake me for an anti-k8s type: I'm not. I'm anti- the foolishness of people trying to bend k8s into something it was never designed to be, to the exclusion of literally everything else in the computing world.

[zarzavat]

> but a lot of uses it's put to seem to be driven more by a desire to have kubernetes on one's resume

It’s no wonder people feel compelled to do this, given how many employers expect experience with k8s from applicants.

Kubernetes is a computer worm that spreads via resumes.

[marcosdumay]

> It was designed to solve "hyper" scale problems

Even then, IMO, it makes too little sense. It would be a bit useful if non-used containers wasted a lot of resources, or if you could get an unlimited amount of them from somewhere.

But no, just creating all the containers you can and leaving them there wastes almost nothing, they are limited by the hardware you have or rent, and the thing clouds can rent are either full VMs or specialized single-application sandboxes.

AFAIK, containers solve the "how do I run both this PHP7 and this PHP8 applications on my web server?" problem, and not much more.

[trueismywork]

Which language do you develop in?

[topspin]

In no particular order; Python, Go, Perl, C, Java, Ruby, TCL, PHP, some proprietary stuff, all recently (last 1-2 years) and in different versions: Java: 8, 11 and 17, for example. Deployed to multiple environments at multiple sites, except the C, which is embedded MCU work.