I'm curious as to why this isn't 0ms for a VM. The entire state of the system can be known ahead of time. Why does the kernel need to do any kind of dynamic initialization? Why aren't all data structures and variables statically assigned to proper values for the given VM. So at VM start up time it simply enters the main dispatch loop.
Mostly because that's not how the kernel is designed. While what you're saying is true, it would require a fundamentally different kernel design. The kernel as it is today is designed to work on a variety of devices, not just VMs, so it behaves in a way in which it doesn't have specific knowledge of the device prior to initialization. To do what you're suggesting, the kernel would need to have a special mode where it supports injection of this information from the VM host in advance of initialization, and there's probably a bunch of possible security issues involved as well.
Well it's not an ordinary boot of course, it's a clone and the state on both side of the fence should be faithfully cloned including virtual devices. The guest kernel will have no way of knowing whether it was resumed once or twice.
Applicability is limited, but works well to spin up serverless compute on-demand.
I wonder the same thing for my regular OS too. Sure the first time it boots on this hardware, it needs to see what's what. But the subsequent hundred/thousand boots, at least for my usage, will be identical.
Before you can reload some previous known state, you need to initialize the hardware, load the firmwares and probe the devices (USB, SATA, PCI), discover the storage volumes and the partition tables. All these steps take time.
That's kind of the point of Windows' Fast Boot option. The base OS pretty much just hibernates at a normal shutdown. No point in re-initializing and restarting all the base services.
There is also often a UEFI option for a "fast boot" as well, where it skips some steps in its initialization, doesn't wait for any user input or logging to the local console if its coming back from a successful boot/power off cycle. But that's separate from the Windows feature which is related to hibernation. You could have Windows Fast Boot enabled while having your UEFI set to a full boot cycle or vice versa.
The UEFI option should work fine with pretty much any OS the hardware should be able to boot.
Exactly, you just need to separate things that need to happen before snapshot from things that must be run every time. Then you just snapshot as part of the build.
This is how GraalVM and OpenJ9 achieve instant startup of Java programs.
"I think the fastest I got the FreeBSD kernel booting in Firecracker was 21 ms. NetBSD is now at 18 ms... I need to go back and address some more of the issues I noticed but didn't get around to fixing.
Anyone know what the current record for Linux is? Last I heard was ~50 ms."
The bug Netflix tripped over was something I introduced while shaving off milliseconds, yes.
The "other issues I didn't get around to fixing" are things like precomputing lookup tables (we can wait and do them on demand, or not at all if they turn out to never get used) and an O(n^2) issue registering names of sysctls.
That's nice, personally it takes me about 400 ms to start a linux kernel with qemu (more time is spent by qemu initializing its network interfaces that starting the kernel and the init!)
I'll see which tricks can be reused on linux, as I'd love to cut that by 1 order of magnitude!
With 1 CPU and 128 MB of RAM. Sure, it's a small system, but it's enough to be useful for some purposes -- more useful than medical discoveries in mice, at least.
"Boot" isn't even a sensible concept in a VM, IMHO, you could just toss an image into RAM. There's no hardware to initialize, the caches are probably already warm, etc.
You do need to shake hands with the virtio devices; that's a bit easier than most hardware, but it's not trivial.
The "take a snapshot after you finish booting and resume that" approach can work (Lambda does it) but it only works if you have the same type and number of CPUs, the same amount of RAM, the same filesystem on disk, and even the same MAC addresses on your network interfaces. So it's not like FreeBSD can ship useful "pre-booted" images.