[kuhsaft]

Fly.io claims it’s “just a VM”. But, Fly.io Machines are an abstraction of microVMs using Firecracker. Building upon that, the FKS implementation is an abstraction on top of Fly.io Machines. So what I’m asking is how, if even, does the FKS implementation support multiple containers for a pod? Using FKS, the abstraction is no longer a VM.

It seems that Fly.io Machines support multiple processes for a single container, but not multiple containers per Machine [0]. This means one container image per Machine and thus no shared network namespace across multiple containers.

[0] https://community.fly.io/t/multi-process-machines/8375

[tptacek]

You can run Docker on a Fly Machine, and run arbitrary numbers of containers inside of it. Or you can run lots of small Fly Machines. FKS is just one model for deploying things.

[kuhsaft]

Right, but what is the point of FKS then? It’s no longer Kubernetes if it doesn’t support a core behavior of Kubernetes.

If you only support deploying single containers with single processes on FKS, then you might as well use flyctl.

It’s a solvable issue of course. The virtual-kubelet implementation would need to create a Machine running a container runtime image that would then run a the pod containers to match the pod configuration.

I think that some disclaimers of the limitations of FKS compared to standardized Kubernetes should be present and highly visible.

[jipperinbham]

We're actively working on the ability to run multiple processes with different images because it's something people using our platform want and it just happens to also be something needed for us to make FKS a more standardized Kubernetes offering.

[kuhsaft]

Seems like Fly.io Machines are trying to reimplement Kata Containers with the Firecracker backend [0], but also abstracting away the host hypervisor machine infrastructure.

Kata has a guest image and guest agent to run multiple isolated containers [1].

[0] https://katacontainers.io/

[1] https://github.com/kata-containers/kata-containers/blob/main...

[tptacek]

The article discusses what you get by using K8s alongside Fly.io. If you want to bin-pack containers onto Fly Machines, you can of course just boot up your own K8s cluster here; that has always been an option.

[kuhsaft]

It should discuss what you don’t get compared to the standard behaviors of Kubernetes.

[remram]

The problem is not the need to bin-pack, the problem is completeness. Sidecars and multiple containers are used for logging, backups, etc. Not to mention that if you grab a manifest or chart for an app, it is going to have pods with multiple containers (whether that's strictly needed or not), and those won't work on fly.io.

This is a critical feature available in every Kubernetes offering, and as such people rely on it. Trying to say that maybe you can do without is missing the forest for the trees.

Like saying that your C compiler doesn't need arrays because it has pointers: sure, maybe, but now good luck compiling any existing code. Maybe don't call it a C compiler if no C program will work on it unmodified.

[tptacek]

Again: Fly Machines are just Linux VMs, and you have root on them.

[remram]

I'm discussing the capabilities of Fly Kubernetes, not Fly Machines. This is good news though, it means they might get there in the future.