see other comment, the author describes some issues with current hardware virtualization. kvm is also pretty good, but not perfect... and completely irrelevant with GPU pass-through enabled. =3
The memory areas would appear as ciphertext to other processes/unprivileged-cores in most cases even when hardware has glitched up. If you are asking how they specifically implemented the mmu <-> unreachable key handling outside the OS, that information was never public if I recall.
I've often pondered how it was really implemented too. Best of luck. =3
"Why Multi-Threaded Code Can Sometimes Misbehave (Weak Memory Concurrency)" (Computerphile)
https://blog.invisiblethings.org/2008/09/02/three-approaches...