[trebligdivad]

Nested virt on x86 is curiously painful; you'd kind of think each layer would be isolated, so that the L0 (hardware) would only have to worry about it's VM (L1), and L1 would have to worry about it's VM (L2); but nope - the L0 top level hypervisor sees faults from the L2 and has to figure out that they are actually L2 not L0. IMHO the extra complexity (and historical flakiness of it) - makes me say that enabling nesting is a bad idea for public VM hosts.

[stinkbeetle]

The alternatives are the L0 emulates a hypervisor-privileged mode to the L1 which would be a performance cost (and does not really avoid the problem since the L0 would have to emulate vmenter/vmrun and therefore be aware of the L2 anyway, or the hardware would have to provide some nested virtualization facility which seems like it would be complicated and expensive for little benefit, though I could be mistaken.

Does anything do a "real" nested virtualization in hardware? s390 might but I know ~nothing about it and it probably does not expose its bare metal hardware layers to Linux/KVM anyway.

[nijave]

IBM mainframes might but I have a very shallow understanding.

Found this https://sixe.eu/news/kvm-nested-vms-ibm-power-linux-lpar

[bonzini]

That's exactly the same as x86. Nested virtualization support is almost entirely in the hypervisor.

[nijave]

x86 doesn't have a firmware hypervisor that allows splitting into LPARs.

That's showing 3 layers of nesting: LPAR, hypervisor, nested hypervisor

[bonzini]

On POWER, LPARs and the first level of hypervisor do not use any LPAR-specific nesting support in the processor. It's all handled by the firmware, not the hardware.

[bonzini]

No, even s390 needs shadow paging.

Intel and AMD both have some small amount of acceleration of nested virtualization, respectively with shadow VMCS and virtualized VMLOAD/VMSAVE.

[iririririr]

but that's the entire point of kvm. that leak is the feature! that's how you get the performance boost.