[sofixa]

> Kubernetes on top of OpenStack on top of Kubernetes (yes, that's what they are doing).

OpenStack's services are running in Kube? And Kube itself is ran as an OpenStack thing? Why? Why not use the same tooling used to deploy that initial Kube to deploy as many as needed? Still a massive maintenance burden, but you don't need to add OpenStack into the mix.

[mrweasel]

Because you can't necessarily run everything in Kubernetes, or in the same cluster. OpenStack probably provides VMs, private networks and bunch of other stuff to run legacy systems, 3rd. party software, Windows application, tons of stuff that can't be containerized.

You can have a large Kubernetes cluster running OpenStack, because it's probably the easiest way to deploy and maintain OpenStack. You then build smaller, isolated Kubernetes clusters on top of OpenStack, using VMs.

It's not as crazy as it sounds, but it does feel a little unnecessarily complex.

[hamandcheese]

I get why you might want to use open stack.

And I get why you might want to use open stack on Kubernetes.

What I don't get is why you would want Kubernetes on open stack on kubernetes.

[fragmede]

My money's on Conway's law. There's a hardware team that's in charge of the hardware, and they need to orchestrate ask the nodes, then there's the openstack team that's their customer who’s in charge of providing a cloud-like environment to the rest of the company including windows VMs, then there’s an applications kube team that provides kube for services that run on kube, with finally kube-ized applications teams that run on the very top.

[mrweasel]

One reason could be that you use Kubernetes as a deployment tool, but you don't actually need to full capacity of three bare metal servers. So you need to slice up the physical servers in some way, and Kubernetes can't do that.

From experience most Kubernetes clusters are actually large enough, in terms of capacity required, to justify using an entire modern server and companies are very reluctant to run a mix if various application on the same cluster. There are very very few organisations large enough to need bare metal servers as Kubernetes worker nodes. Unless you use them to run OpenStack.

[derefr]

From what I've seen in other projects, I think that translates to:

1. we have a management k8s cluster where we deploy app blueprints

2. the app blueprints contain, among other things, specifications for VMs to allocate, which get allocated through an OpenStack CRD controller

3. and those VMs then get provisioned as k8s nodes, forming isolated k8s clusters (probably themselves exposed as resource manifests by the CRD controller on the management cluster);

4. where those k8s nodes can then have "namespaced" (in the Linux kernel namespaces sense) k8s resource manifests bound to them

5. which, through another CRD controller on the management cluster and a paired CRD agent controller on in the isolated cluster, causes equivalent regular resource manifests to be created in the isolated cluster

6. ...which can then do whatever arbitrary things k8s resource manifests can do. (After all, these manifests might even include deployments of arbitrary other CRD controllers, for other manifests to rely upon.)

All said, it's not actually that braindead of an architecture. You might better think of it as "k8s, with OpenStack serving as its 'Container Compute-Cluster Interface' driver for allocating new nodes/node pools for itself" (the same way that k8s has Container Storage Interface drivers.) Except that

1. there isn't a "Container Compute-Cluster Interface" spec like the CSI spec, so this needs to be done ad-hoc right now; and

2. k8s doesn't have a good multi-tenant security story — so rather than the k8s nodes created in these VMs being part of the cluster that spawned them, their resources isolated from the management-layer resources at a policy level, instead, the created nodes are formed into their own isolated clusters, with an isolated resource-set, and some kind of out-of-band resource replication and rewriting to allow for "passive" resources in the management cluster that control "active" resources in the sandboxed clusters.