[cimnine]

Commercial HSM have ways of exporting the key they hold onto smartcards. Usually the keys are split onto a number of smartcards, let's say 3. For increased robustness, each third is written to two smartcards. (We now have 6 cards.) These smartcards each belong to one person, who is the only one knowing the PIN that protects the third of the key it holds. Each of these smartcards is then brought to different bank vault, sealed in tamper-proof bags.

To restore the key, you need to bring 3 out of 6 persons to the so-called key ceremony, and each has to bring his smartcard and his PIN.

The same mechanism can be used to provision multiple HSMs with the same key material. But there are other means to do this. As soon as two HSM share a common secret, also known as Key Sharing Key, they are able to exchange all key material they possess in a secure manner.

Some HSM don't even bother to store the keys they generate within the bounds of their hardware. Only it's master key is stored in it's hardware, any other key is encrypted with the master key and stored on a shared filesystem.

If this sounds artificial to you, let me assure you that such procedures are in place at various companies who deal with raw credit card data, at least in Europe. The EMV committee, the PCI organization and each issuer of credit card do mandate such procedures.

And they are very strict. We once had to ship HSMs back to the vendor, because at some point they were not supervised by at least two persons. (At least the documentation thereof was missing.)

[cimnine]

... because at some point they were not supervised by at least two persons, before it was taken into operation, that is. (Afterwards, they have to be locked into a rack that requires two different badges to open it's doors and which must have a CCTV system recording it at all times.)