[crabbone]

I don't know the details of this particular migration, but I used to have a VM in some low-price tier that was running for a long time (few years), and, eventually AWS sent me an email telling they are going to shut it down for maintenance reasons.

Guess this was something similar. VMs, if not specifically configured to be able to move cannot really be moved automatically. Think about eg. randomness of ordering in PCIe bus (i.e. after moving the devices may not come up in the same order as before moving), various machine ids, like MAC address -- if you don't make sure VM isn't affected by these changes, it's likely that it will be, if moved.

[mschuster91]

> Think about eg. randomness of ordering in PCIe bus (i.e. after moving the devices may not come up in the same order as before moving), various machine ids, like MAC address -- if you don't make sure VM isn't affected by these changes, it's likely that it will be, if moved.

QEMU/KVM/libvirt/... are idempotent when it comes to hardware the VM sees - the exception is the CPU model, that one can't be changed on the fly without at least rebooting the VM in question, and hardware in passthrough mode like GPUs.

All the VM sees from a live migration is a few seconds of "lost" time, as if someone had stopped the CPU clock.

[ec109685]

The clouds have figured this out: https://dl.acm.org/doi/pdf/10.1145/3186411.3186415

[crabbone]

That's a very bold claim :)

If you prepare for migration, then it will work. If you don't -- it might or might not work, and it depends on way too many things to be confident that it will.

For example, in our VM deployment process we heavily rely on PXE boot and our code that runs during initramfs and also after pivot. So, even if whatever you have in the hypervisor and the virtualized OS has somehow managed to solve the moving problem, our own (screwy) code might not be so smart.

In general, the more of the underlying infrastructure you touch, the harder it will be to move you, unless you specifically prepare to move. Eg. what if you set up BMC for various components on your system? What if your code relies on knowing particular details of hardware to which you were given pass-through access from your VM, like, eg. you record somewhere the serial number of your disk in order to be able to identify it on next boot, but suddenly that disk is gone?

Even simpler: MIG flag on NVidia's GPUs is stored in the persistent writable memory of the GPU itself. Suppose your VM has moved to a different location and is connected to a different (but compatible) GPU -- you need to run the code that sets the flag again and then reboot the system in order to start working with the GPU, but the host may not be even aware of the fact that you need that particular setting.

The guest side of things needs to be prepared to move, to mitigate these problems.

[ec109685]

I'm saying that at least GCP has supported VM migration for a while and it's generally not something people are currently worried too much about these days given they have attempted to mitigate issues like you've pointed out.