[parasense]

> Musk fired everyone with access to the private key to their internal root CA,

The way forward is to generate a new CA root certificate.

> and they can no longer run puppet because the puppet master's CA cert expired

They can reconfigure internal tools to use the new CA root certificate, or rather one of the signed intermediate certificates.

> and they can't get a new one because no one has access.

They can simply generate new CA root certificates, and sign or create new intermediate certificates.

> They no longer can mint certs.

Yes, they, can...

> My limited understanding in this area is that this is...very bad

No, it, is, not...

There are two immediate issues that come to mind.

* Twitter was so awful before, that it relied on people to safeguard the keys to the kingdom. This is very bad practice, and one of the many things Musk will no doubt be fixing. For any mission critical assets, and especially certificates, but also passwords... current modern day corporate practice is to have a secure ledger of these that can be accessed by the board of directors, the executive managers, and designated maintainers. At no point ever should the password be entrusted to anybody, but rather a "role" that functions as the one who has access. Say for example, the CIO/CTO and their subordinates.

* The Second issue is the one everyone is fixating upon, and that's firing important people who put the company at risk. This is a big issue, and certainly Musk could have done a better job of scoping out who represents a single-point-of- failure at twitter, eliminate that risk, and then proceed with the culling. In a modern enterprise no single person should be capable of putting the entire operation at risk. It's just that simple. So in a way, Musk accelerated what was probably inevitable at Twitter already. They were probably precariously close to destruction already, and now they can learn the hard way of not repeating these mistakes.

[spaced-out]

>Twitter was so awful before, that it relied on people to safeguard the keys to the kingdom. This is very bad practice...that can be accessed by the board of directors, the executive managers, and designated maintainers.

LOL, you realize all the PEOPLE you list as the PEOPLE who should be able to manage the keys to the kingdom are PEOPLE? Board of directors - fired on day one of Musk takeover, executive managers - many fired one day one by Musk as well, designated maintainers - for all we know they could have been fired in the purge or quit when Musk offered the 3 month severance.

All system require people to run.

[MichaelZuo]

According to the latest publicly available sources there are still many hundreds of folks on the active payroll at Twitter, do you know of any evidence to the contrary?

[Volundr]

Significant edits for clarity.

Serious question... How do I build a system that grants access to a company role not a person? In other words, the CIO is fired, how does this system ensure that the new CIO can access it, and the old one no longer can?

If we tie it to the HR system, whoever admins that effectively has the keys to the kingdom. Same for Active Directory or any other technical solution.

[caltheon]

Something like the nuclear football is probably the only answer. Something very obvious that is transferred with the role

[Volundr]

You're probably right, though honestly I'm not sure that helps here either. If I'm the CIO and Musk walks in and tells me to get out, I'm not going to go to any pains to make sure he knows about the football. Sure I'll leave it there in my desk, where if someone knows of it's existence they can find it, but it probably just ends up going in the dumpster or with the desk when he sells it.

[ryan_lane]

> For any mission critical assets, and especially certificates, but also passwords... current modern day corporate practice is to have a secure ledger of these that can be accessed by the board of directors, the executive managers, and designated maintainers. At no point ever should the password be entrusted to anybody, but rather a "role" that functions as the one who has access. Say for example, the CIO/CTO and their subordinates.

Maybe in hacker movies. In real life, you try your best to avoid anyone having access to keys or passwords, and rely on HSMs, cloud KMS, secret services, etc. Access to those things is controlled by your security team, with multi-factor authentication, often stored in safes, with alerts being fired when they are used (because they should never be used). The audit logs that trigger these alerts should be written in WORM storage, so you can track access back down to individuals, and so that you know when you need to rotate secrets accessed by humans. Ideally your CA infrastructure automatically rotates and distributes.

There's absolutely no way in hell you should allow your board to have access to these things.

Most companies slowly work their way towards full automation, and until that happens, your security team usually owns manual rotations of critical systems like this. Only a fucking moron would fire all of these people.

[foolfoolz]

when building secure systems one of the key principles is assume someone will leak the private key. this is how we get to hsm

maybe another one is assume you will lose access to the hsm. sure spinning up a new trust chain is annoying but it wouldn’t take that long to do. totally agree this post is overblown

[anecdotal1]

spinning up a new trust chain is not so hard, but deploying that trust chain to thousands of servers around the world when your automation tool isn't available to do it with is really, really hard.

[lamontcg]

This is why I've been very skeptical of the kids these days kicking literally everyone off of the production servers.

Having a few greybeards with the keys to the kingdom and the wisdom not to use it to screw around in prod, outside of existential emergencies, can be quite useful.

Also should have console access.

One time a bad config push took out a couple hundred webservers with effectively a single iptables default deny rule and we had to get a dozen people to fix them in chunks by logging in manually over remote terminal (probably could have expect-scripted that up, but it was quicker to just get it done).