[kuhsaft]

How does this handle multiple containers for a Pod? In a container runtime k8s, containers within a pod share the same network namespace (same localhost) and possibly pid namespace.

The press release maps pods to machines, but provides no mapping of pod containers to a Fly.io concept.

Are multiple containers allowed? Do they share the same network namespace? Is sharing PID namespace optional?

Having multiple containers per pod is a core functionality of Kubernetes.

[remram]

You can use mount namespaces, or even containers in your VM. Maybe that's how?

[kuhsaft]

Fly.io claims it’s “just a VM”. But, Fly.io Machines are an abstraction of microVMs using Firecracker. Building upon that, the FKS implementation is an abstraction on top of Fly.io Machines. So what I’m asking is how, if even, does the FKS implementation support multiple containers for a pod? Using FKS, the abstraction is no longer a VM.

It seems that Fly.io Machines support multiple processes for a single container, but not multiple containers per Machine [0]. This means one container image per Machine and thus no shared network namespace across multiple containers.

[0] https://community.fly.io/t/multi-process-machines/8375

[tptacek]

You can run Docker on a Fly Machine, and run arbitrary numbers of containers inside of it. Or you can run lots of small Fly Machines. FKS is just one model for deploying things.

[kuhsaft]

Right, but what is the point of FKS then? It’s no longer Kubernetes if it doesn’t support a core behavior of Kubernetes.

If you only support deploying single containers with single processes on FKS, then you might as well use flyctl.

It’s a solvable issue of course. The virtual-kubelet implementation would need to create a Machine running a container runtime image that would then run a the pod containers to match the pod configuration.

I think that some disclaimers of the limitations of FKS compared to standardized Kubernetes should be present and highly visible.

[jipperinbham]

We're actively working on the ability to run multiple processes with different images because it's something people using our platform want and it just happens to also be something needed for us to make FKS a more standardized Kubernetes offering.

[kuhsaft]

Seems like Fly.io Machines are trying to reimplement Kata Containers with the Firecracker backend [0], but also abstracting away the host hypervisor machine infrastructure.

Kata has a guest image and guest agent to run multiple isolated containers [1].

[0] https://katacontainers.io/

[1] https://github.com/kata-containers/kata-containers/blob/main...

[tptacek]

The article discusses what you get by using K8s alongside Fly.io. If you want to bin-pack containers onto Fly Machines, you can of course just boot up your own K8s cluster here; that has always been an option.

[kuhsaft]

It should discuss what you don’t get compared to the standard behaviors of Kubernetes.

[remram]

The problem is not the need to bin-pack, the problem is completeness. Sidecars and multiple containers are used for logging, backups, etc. Not to mention that if you grab a manifest or chart for an app, it is going to have pods with multiple containers (whether that's strictly needed or not), and those won't work on fly.io.

This is a critical feature available in every Kubernetes offering, and as such people rely on it. Trying to say that maybe you can do without is missing the forest for the trees.

Like saying that your C compiler doesn't need arrays because it has pointers: sure, maybe, but now good luck compiling any existing code. Maybe don't call it a C compiler if no C program will work on it unmodified.

[javaunsafe2019]

Why should you do this - sounds like an antipattern to me

[kuhsaft]

It’s used widely in the Kubernetes world and is known as sidecars [0].

[0] https://kubernetes.io/blog/2023/08/25/native-sidecar-contain...

[javaunsafe2019]

Widely? I lived in this helm, kubernetes, pulumi world the past 4 years and we followed the simple rule of one service/container per pod. Why add complexity where it’s not needed. Like running a dB a and a service in the same docker container - a no go for me and many.

[ossm1db]

Kubernetes official documentation states, "A Pod is a group of one or more containers, with shared storage and network resources, and a specification for how to run the containers."

Running more than one container in a pod is a fundamental concept of Kubernetes. Init containers and sidecars allow for a separation of concerns, which is essential for non-cloud-native workloads. Logging and telemetry are just a couple of features which may be designed and built into cloud-native applications, but legacy applications need this flexibility without modifying the application itself.

The fact you ran K8S for four years without it demonstrates only that it is not required by your workload -- not that it is "unnecessary complexity" or an "anti-pattern."

[kuhsaft]

The thing is you might not be adding the sidecars yourself.

Kubernetes has a resource called MutatingAdmissionWebhook that allows mutations of the Pod object before creation.

I think the most common use case is for service meshes. Your Deployment might not have any sidecars, but the service mesh controller will automatically add a network proxy container to your pod via the admission webhook.

Another use case would be OpenTelemetry or some other observability service sidecar injection for auto-instrumentation.

[javaunsafe2019]

Ok that was something inward missing on my end. Thanks for all the explanations and sorry for me being wrong

[qeternity]

Nobody would suggest this. But what about a metrics scraper for your db pod? That’s where sidecars come in.

[GauntletWizard]

Like helm, it's a widely used anti-pattern

[postalrat]

Its because you have multiple processes (containers) that work together in their little pod. You could stick them all in a single image somehow but that would be much more work and less flexible.