[edude03]

I'm confused about what this is actually offering (also very tired due to some flight problems; anyway)

To me, I'd imagine kubernetes on fly as running kind (kubernetes in docker) with fly converting the docker images to firecracker images OR "normal" kubernetes api server running on one machine then using CAPI/or a homegrown thing for spinning up additional nodes as needed.

So, what's the deal here? Why k3s + a virtual kublet?

[tptacek]

You can certainly boot up your own K8s cluster, any way you'd like to, just by enlisting a bunch of Fly Machines and configuring them yourself. A Fly Machine is just a VM, and you have root in the VM. You can set up systemd, you can set up Docker, you can run kubelets on all your Machines.

The thought here is: Fly.io already does a lot of the things any K8s distribution would do. If you were to boot up a complete K8s distribution on your own Fly Machines, running oblivious to the fact that they were on Fly.io, you'd be duplicating some of the work we'd already done (that's fine, maybe you like your way better, but still, bear with me).

So, rather than setting up a "vanilla" K8s that works the same way it would if you were on, like, Hetzner or whatever, you can instead boot up a drastically stripped down K8s (based on K3s and Virtual Kubelet) that defers some of what K8s does to our own APIs. Instead of a cluster of scheduling servers synchronized with Raft, you just run a single SQLite database. Instead of bin-packing VMs with Docker and a kubelet, you just run everything as an independent Fly Machine.

We took the time to write about this because it was interesting to us (I think we expected a K8s to be more annoying for us to roll, and when it was easier we got a lot more interested). There are probably a variety of reasons to consider alternative formulations of K8s!

[edude03]

Right, I think I get it now, like benpacker said in https://news.ycombinator.com/item?id=38685760 it's a way to map the kubernetes API to the fly platform (basically). Which makes a lot of sense!

Not sure what the implications of that are in practice but sounds interesting.

[candiddevmike]

Did you consider using/adopting the seemingly defunct nomad virtual kubelet?

[tptacek]

Do people want to run Nomad on Fly.io? It wouldn't have addressed any of the reasons we replaced Nomad with flyd; it's hard to scale a globally synchronized distributed database for real-time scheduling.

[candiddevmike]

I thought you guys used Nomad, disregard. The virtual kubelet for Nomad would've let you proxy Kubernetes into your Nomad environment basically.