[dboreham]

Interested to hear more about your use case and threat model, if you are willing to share. I ask because although I've looked into (and done some prototyping) with secure cloud hosting, I/we came to the conclusion that there's no current technology that is "actually secure" and so abandoned the approach. Curious if things have improved now, or if you're operating in some security theater context where it's ok.

[manawyrm]

+1, if your threat model is actually this severe, use physical hardware with physical interlocks and physical security mechanisms.

Software/virtualization is just helpless against such a threat model.

[Qasaur]

The basic principle is to ensure that any machine/workload which joins the network (and processes customer data, in this case extremely sensitive PII) has a cryptographically verified chain of trust from boot to the application-layer to guarantee workload integrity.

NixOS is used for declarative and more importantly deterministic OS state and runtime environment, layered with dm-verity to prevent tampering of the Nix store. The root partition, aside from whatever is explicitly configured in the nix store, is wiped on every reboot. The ephemerality prevents persistence of any potential attacker, and the state of the machine is completely identical to whatever you have configured in your NixOS configuration, which is great for audibility. This OS image + boot loader is signed with organisation-private keys, and deployed to machines preloaded with UEFI keys to guarantee boot integrity and preventing firmware-level attacks (UEFI secure boot).

At this point you need to trust the cloud provider to not tamper with the UEFI keys or otherwise compromise memory confidentiality through a malicious or insecure hypervisor, unless the provider supports memory encryption through something like AMD SEV-SNP. The processor provides an AMD-signed attestation that is provided to the guest OS that states "Yes, this guest is running in a trusted execution environment, and here are the TPM measurements for the boot" and you can use this attestation to determine whether or not the machine should join your network and that it is running the firmware, kernel, and initramfs that you expect AND on hardware that you expect.

I think I'll put together a write-up on this architecture once I launch the service. There is no such thing as perfect security, of course, but I think this security architecture prevents many classes of attacks. Bootkits and firmware-level attacks are exceedingly difficult or even impossible with this model, combine this with an ephemeral root filesystem and any attacker would be effectively unable to gain persistence in the system.