How would this compare for current users of Docker for Mac? If I use Docker for Mac to run two containers, I get one VM that runs two containers, right? And if I use this, then I get two VMs, one per container?
How does the resource usage compare?
I have to tell Docker for Mac how much RAM to allocate, and that's then shared between the containers. Can krunvm use as much RAM as the container needs, or does it also need some pre-allocation?
I can't comment on the comparison with Docker for Mac bacause, honestly, I have never used it.
With krunvm, each session initiated with "krunvm start" is an independent lightweight VM. The maximum amount of RAM the VM can use is configured with the "mem" flag (or it uses what's configured by default), but the VMM will always try to use the minimum possible amount of RAM by returning the pages the guest is no longer using to the host (virtio-balloon's free page reporting feature).
That said, you can also start a single VM (using "krunvm start"), run podman's service ("podman system service...") inside it, switch to another terminal in macOS and execute multiple containers inside the VM using "podman remote" [1]. Now I'm thinking I should probably write a tutorial about this option ;-)
krunvm uses libkrun (https://github.com/containers/libkrun) for executing the VM, and while the later is also based in rust-vmm and shares some code with Firecracker and Cloud-Hypervisor, it's specialized in the process isolation use case. This means it implements a different set of devices (most notably, virtio-fs instead of virtio-blk, and virtio-vsock+TSI (Transparent Socket Impersonation) instead of virtio-net), and it takes the form of a dynamic library instead of a final binary.
In fact, the networking limitations are caused by this use of virtio-vsock+TSI. TSI (WIP implementation here: https://github.com/containers/libkrunfw/blob/main/patches/00...) is an experimental mechanism that provides inbound and outbound networking capabilities to the guest, with zero-configuration and minimal footprint, by transparently replacing user-space AF_INET sockets with AF_TSI, that have both an AF_INET and AF_VSOCK personality.
TSI has the additional advantage that, for the host side, all connections appear to come and go to the process acting as a VMM (in this case, krunvm, as it links directly with libkrun), which makes it very container-friendly in a way that even side-cars (such as Istio) work out-of-the-box.
Also, firecracker did not merge virtio-fs [1]. So I guess it is a scope and priority decision:
> To not trade off our core use case simplifying assumptions, we'll need something like a way to build different Firecracker "specialized variants" or alternatively a plugin system. This is something worth discussing for 2021. Again this is not a guarantee, but virtio-fs could fit in such a scheme, especially if it's a rust-vmm crate we can easily import over.
I’m honestly just curious how they managed to get a KVM based solution working on MacOS ARM64. Buildah and few other dependencies appeared to be Linux only... but the brew install worked..
The key is that libkrun (https://github.com/containers/libkrun), the library that krunvm uses for running the VMs, as recently integrated support for Hypervisor.framework on ARM64, in addition to KVM.
How does the resource usage compare?
I have to tell Docker for Mac how much RAM to allocate, and that's then shared between the containers. Can krunvm use as much RAM as the container needs, or does it also need some pre-allocation?