The memory areas would appear as ciphertext to other processes/unprivileged-cores in most cases even when hardware has glitched up. If you are asking how they specifically implemented the mmu <-> unreachable key handling outside the OS, that information was never public if I recall.
I've often pondered how it was really implemented too. Best of luck. =3
"Why Multi-Threaded Code Can Sometimes Misbehave (Weak Memory Concurrency)" (Computerphile)
Thus, writing a robust and secure OS may actually be possible by competent programmers in most compiled languages. Best of luck =3