Could you elaborate more on the control mechanisms that supported the outcome of "your machine will often fail to boot, can't be remotely boot-unlocked" for said anti-supported use cases?
As far as I was able to tell, the system didn't know which NIC it should be routing through. So it would sit at the disk encryption screen and you wouldn't be able to remote unlock it. And you don't have access to the building to go in and do it in person. Or if you successfully cleared that, the OS also rolled the dice to let you authenticate. Rebooting was painful, typically taking 5-15 minutes. And if you tried to edit configurations to try and load the dice, puppet would overwrite them. Tickets seeking help were mostly ignored, citing that dual NIC was explicitly not supported.
That's interesting to hear that you can remote-unlock machines that are waiting for a disk encryption key. I presume the remote-unlock case is the primary one.
I've long been curious how early boot works on Google servers (and TIL workstations too, although it makes perfect sense) - primarily because I want to copy the techniques myself! :D
How is key storage and device attestation actually done?
In reality, what we can expose, is that during the early boot process, our systems reach out to another system that register's its interest, and as a user (from another trusted device, phone, laptop, etc.) you can visit the web service and click a button.
How and where keys are stored is a great big "?" that's up to the implementer to solve..
So basically you have 2FA for workstation boot. That's really quite cool.
Of course I want to know how and where the keys are stored :D so I can do the same thing myself! Arguably security systems in this class demonstrate their integrity *because* their architecture is fully open, documented and straightforwardly reproducible. It isn't science if it isn't reproducible, right? Something something computer science...
That's the idealistic view, of course. In the <insert cartoon punching fight cloud here> real world, we have The Legacy PC Problem™, where secure boot isn't, TPMs can be bus sniffed, SGX doesn't really support the hacker/tinkerer exploration necessary to power defense in depth, ME is a ginormous black box that eats authentication headers like they might as well be glue... and it doesn't matter that I am a dog residing on Mars because the MDM my 2FA device is signed into has decided I'm legit.
Hence my interest in real security. It's a giant debacle, surely there are some genuinely cool wins to be had out there that truly make a dent ._.
The normal non-Google way is to configure initramfs to start an SSH server on a non-default port, and then you log in (from your secure workstation with a dedicated public/private key, of course) and use systemd-tty-ask-password-agent to enter the disk encryption key.
You can then monitor whether your service is up, and if you get a notification that it's down use the hosting service's management interface to reboot the machine/VM and then do the SSH thing.