[drewg123]

Without any context in terms of what the VM is doing or what VMM software you use, my best guess is that the OS/VMM are pre-allocating memory for the VM. This might involve paging out other processes' memory, which could take some time.

I think task manager would tell you if there is a blip of memory usage and paging activity at the time. And I'm sure windows itself has profilers that can tell you what is happening when the VM is started..

[dataflow]

VirtualBox on Windows, primarily. Though I feel like haven't seen other VMs in the past start up a whole ton faster (maybe a somewhat) (ignoring WSL2). Page files are already disabled, there's plenty of free RAM, and it makes no difference how little RAM the guest is allocated (even if it's 256MB). So no, those are not the issues. VirtualBox itself seems to be doing something slow during that time and I don't know what that is.

[gopher_space]

I remembered something about VirtualBox not playing nicely with Hyper-V on Windows, and dug up a possibly relevant post[0] on their forums. IIRC we ended up moving a few build systems to Docker and dropping VirtualBox because of hyper-v related issues, but it's been a few years.

[0] https://forums.virtualbox.org/viewtopic.php?t=112113

[dataflow]

That's the unrelated green-turtle issue. It's only relevant after the guest has actually started running instructions. I'm talking about before that point.

[gopher_space]

I'm not aware of any turtles, that was just the first thing I found when trying to see if VirtualBox and Hyper-V were still a problematic combo.

Again, it was a few years ago, but we didn't solve the problem or identify an actual root cause. We stopped banging our heads against that particular wall and switched technologies.

[mgerdts]

What is your definition of free memory? If the system has read a lot of data, the page cache is probably occupying most of the RAM you consider free. Look at cache and standby counters.

I’ve noticed that windows can only evict data from the page cache at about 5 GB/s. I do not know if this zeros the memory or that would need to be done in the allocation path.

A couple years ago I tracked down a long pause while starting qemu on Linux to it zeroing the 100s of GB of RAM given to the VM as 1 GB huge pages.

These may or may not be big contributors to what you are seeing, depending on the VM’s RAM size.

[mynameisvlad]

So the issue is pretty clearly with VirtualBox itself, but you are making it sound like it's an issue with VMs on Windows or in general.

[drewg123]

For some reason I can't reply to your reply. I'd strongly suggest that you profile virtual box. It beats speculation..

[HumanOstrich]

I experienced something similar back when Microsoft decided to usurp all hypervisors made for Windows and make Windows itself run as a VM on Hyper-V running as a Type 1 hypervisor on the hardware. That made it so other VMs could only run on Hyper-V alongside Windows or with nested virtualization.

So this meant VMWare, VirtualBox, etc as they were would no longer work on Windows. Microsoft required all of them to switch to using Hyper-V libs behind the scenes to launch Hyper-V VMs and then present them as their own (while hiding them from the Hyper-V UI).

VirtualBox was slow, hot garbage on its own before this happened, but now it's even worse. They didn't optimize their Hyper-V integration as well as VMWare (eventually) did. VMWare is still worse off than it was though since it has to inherit all of Hyper-V's problems behind the scenes.

Hope this brings some clarity.