[deserialized]

Definitely neat and a huge flex but I think if you want to actually do multi-tenant kubernetes in kubernetes you should be using kubevirt or metal3 to bootstrap a VM and get a real kernel in there to isolate the cluster from its neighbors.

[olivermuty]

It scares me that I have no idea if this is a real comment or just satire :D

[deserialized]

It's real, kubevirt is a kubenetes wrapper for QEMU which allows you to use kubernetes the manage the lifcycle of virtual machines the same way you manage a pod.

Metal3 is just a wrapper for Ironic from open-stack.

The two work together to re-image bare metal machines and attach them to clusters as nodes which can then be sliced up into smaller virtaul machine nodes to sell to your end user which they can then use to run docker, kubernets or whatever else they want.

Doing this allows you to have full kernel level isolation on a per-tenant basis which is not possible just using normal containers.

It also allows you to pin workloads to specific CPU cores for latency sensitive tasks as well as pass pci express devices through to individual VMs on the host and other fun things

It's really only useful if your problem is 'i have one really big computer but I want to have lots of small computers instead' but that's pretty much every data center so shrug

ClusterAPI (CAPI) enters the equation in that CAPI requires Virtual Machines or Bare Metal machines as inputs which it then uses to create a kubenrtes cluster. Thus, kubernetes in kubernetes with kernel level isolation on a per-tenant basis