[iostream24]

“So you want to run a bunch of stuff on one computer, why?”

In a quest to get closer to the metal, Kubernetes keeps you far away, which is the opposite of what any production service should want.

What is the purpose of adding layers when uni-kernels and eco-kernels give you better isolation and better performance?

Your cloud provider already runs your virtual machines OS on a hardware hypervisor. Then running Kubernetes on top of the OS and then a zillion containers on it is a recipe for poor performance.

What is the logic here that I am clearly missing?

Cloud providers native Kubernetes stacks don’t improve performance or pricing, compared to their cloud compute instances virtual machines and a dedicated virtual machine per would-be-container that thankfully doesn’t need to now share processing resources with others.

What gives? Why on earth would anyone run production processes in Kubernetes?

[iostream24]

Merely making scriptable infrastructure doesn’t require kubernetes or containers… Ansible or it’s ilk will do just fine. Deploy cloud instances like you currently deploy containers. Save money, get better performance, have real networking.

*i have used kubernetes and understand how to use it as intended. I feel like I am missing the motivation for it’s current widespread usage in web apps

[CHY872]

The advantage of K8s isn't really in the virtualization technique used, but in the orchestration it can be made to perform for you. You can for sure configure K8s to use a host per container, if this is what you want.

Example of thing that is pretty straightforward in K8s and much less straightforward outside of it.

1. For compliance reasons, you need to make sure that your underlying OS is patched with security updates.

2. This means you need to reboot the OS every X time.

3. You want to do this without downtime.

4. You have a replicated architecture, so you know you have at least two copies of each service you run (and each can handle the required traffic).

In K8s, this operation can be as simple as:

1. Mark your old nodes as unschedulable.

2. Drain your old nodes (which will grab new nodes from your cloud provider, installing the updated machine image).

3. Delete your old nodes.

The exact steps will differ based on your use case, but that's basically it.

Steps you didn't need to think about here:

1. If I'm about to take a node down, do I have enough redundancy to handle its loss? K8s understands the health of your software, and you have a policy configured so it understands if taking down a container will cause an outage (and avoid this). Note: third party tech will be naturally compatible - if you use Elastic's cloud-on-k8s operator to run ES, it'll appropriately migrate from host to host too, without downtime. Likewise, the same script will run on AWS, Azure, GCP.

2. How fast can I run this? If building this logic yourself, you'll probably run the upgrade one node at a time so as to not have to think about the different services you run. But if it takes 15 minutes to run a full upgrade, you can now only upgrade 100 hosts each day. K8s will run whatever it can, as soon as it can without you having to think about it.

3. What happens if concurrent operations need to be run (e.g. scale-up, scale-down)? With K8s, this is a perfectly reasonable thing to do.

4. Does this need to be monitored? This is a fairly standard K8s workflow, with most components identical to standard scale-up/scale-down operations. Most components will be exercised all the time.

Generally I've been impressed by how straightforward it's been to remove the edge cases, to make complex tech fit well with other complex tech.

A while back we upgraded between two CentOS versions. In such a case it's recommended to reinstall the OS - there's not a clear upgrade path. In K8s, this would have been the same set of steps as the above. In many orgs, this would be a far more manual process.

[jiggawatts]

It deduplicates the kernel memory and system image base disk.

The minimum virtual machine size for a Windows server that is at all useful for anything is 4 GB of memory. Okay, okay, so you can technically boot it up on 2 GB and some roles will work fine, this will last only until some dingbat remotes to it with RDP with a 4K monitor and it starts swapping to disk.

Even if you use Server Core and block port 3389, it still needs a ton of memory just to start.

Running in a container it uses a few hundred megabytes.

Similarly, the minimum system disk size you can get away with is 32 GB if it is a discardable / ephemeral instance. You need 64 GB minimum if you ever intend to run Windows Update on it.

With containers, the unique parts of the image might be just a few hundred megabytes, even for complex apps.

My experience is with Windows, but from what I hear Linux VMs vs Linux containers have vaguely similar ratios.

So with containers, a single host can run dozens of applications, all sharing the same base disk, and all sharing the same OS kernel. The savings can be staggering.

At $dayjob, the admins are very much stuck in the dedicated VMs for every role mentality, and they're burning through enormous piles of taxpayer money to run them at literally 0.1% load.

Having said that, Kubernetes has its own problems. As you said, layering it on top of cloud VMs is a bit silly, and can easily result in the container running in a nested hypervisor at molasses speeds. Similarly, every single typical process changes dramatically: Deployment, updates, monitoring, auditing, etc...

Combine the above with the incompatible underlying cloud layer and things get really messy really quickly.

In my experience 90% of the world just isn't ready for the learning curve. Windows as an operating system isn't ready, certainly. Microsoft Azure isn't really ready either. Their AKS managed offering is still undergoing massive churn and seems to have more preview features than stable features. Even in the Linux world I hear more horror stories than success stories. It seems that everyone who says they love Kubernetes is using it on like... one machine. Come back and tell me how you feel after troubleshooting a failed upgrade on a cluster managing $100M of finance transactions.

What I would like to see is "native Kubernetes clouds" where the hosts are bare metal and there is no impedance mismatch between K8s and the cloud provider APIs because K8s is the API. Instead of the Azure portal or the AWS console you literally log into a Kubernetes console.

IMHO that would allow a true commoditisation of the public cloud and start to erode the near-duopoly of AWS and Azure.

[iostream24]

I think exokernels and isokernels solve many of these issues where containers are currently used, check the Ocaml community for examples.

They run on hardware.

Ultimately, there needs to be a singular scheduling system running on hardware and a singular HAL-like driver layer, and exo or iso kernels deliver just that, vs lxe containers provided by os services.

The sizes are also quite impressive.