[foolfoolz]

when building secure systems one of the key principles is assume someone will leak the private key. this is how we get to hsm

maybe another one is assume you will lose access to the hsm. sure spinning up a new trust chain is annoying but it wouldn’t take that long to do. totally agree this post is overblown

[anecdotal1]

spinning up a new trust chain is not so hard, but deploying that trust chain to thousands of servers around the world when your automation tool isn't available to do it with is really, really hard.

[lamontcg]

This is why I've been very skeptical of the kids these days kicking literally everyone off of the production servers.

Having a few greybeards with the keys to the kingdom and the wisdom not to use it to screw around in prod, outside of existential emergencies, can be quite useful.

Also should have console access.

One time a bad config push took out a couple hundred webservers with effectively a single iptables default deny rule and we had to get a dozen people to fix them in chunks by logging in manually over remote terminal (probably could have expect-scripted that up, but it was quicker to just get it done).