[paganel]

This was particularly interesting (from the reddit thread posted above):

> A colleague is dealing with a particularly nasty case. The server storing the BitLocker recovery keys (for thousands of users) is itself BitLocker protected and running CrowdStrike (he says mandates state that all servers must have "encryption at rest").

> His team believes that the recovery key for that server is stored somewhere else, and they may be able to get it back up and running, but they can't access any of the documentation to do so, because everything is down.

[lordnacho]

> but they can't access any of the documentation to do so, because everything is down.

One of my biggest frustrations with learning networking was not being able to access the internet. Nowadays you probably have a phone with a browser, but back in the day if you were sitting in a data room and you'd configured stuff wrong, you had a problem.

[JCharante]

Isn’t that what office safes are for? I don’t know the location, but all the old guard at my company knew that room xyz at Company Office A held a safe with printed out recovery keys and the root account credentials. No idea where the key to the safe is or if it’s a keypad lock instead. Almost had to use it one time.

[sobellian]

Just hope there is no mutual recursion, i.e. recovery key A is stored on machine B, recovery key B is stored on machine A!

[radiator]

I find that hilarious

[lukan]

Me too, as I am also not affected. But I do pity those guys who now try to solve that deadlock.

[RobotToaster]

Nobody, not one person, thought that documentation should be stored in hard copy?

[whynotmaybe]

I'm guessing someone somewhere said that "it must be stored in hard copy in a safe" and the answer was in the range of "we don't have a safe, we'll be fine".

Or worse, if it's like where I worked in the past, they're still in the buying process for a safe (started 13 months ago) and the analysts are building up a general plan for the management of the safe combination. They still have to start the discussions with the union to see how they'll adapt the salary for the people that will have to remember the code for the safe and who's gonna be legally responsible for anything that happens to the safe. Last follow-up meeting summary is "everything's going well but we'll have to modify the schedule and postpone the delivery date of a few months, let's say 6 to be safe"

[KineticLensman]

Not just financial / process barriers. I worked for a company in the early 90's that needed a large secure safe to store classified documents and removable hard drives. A significant part of the delay in getting it was figuring out how to get it into the upstairs office where it would be located. The solution involved removing a window and hiring a crane.

When we later moved to new offices, somebody found a solution that involved a 'stair-walking' device that could supposedly get the safe down to the ground floor. This of course jammed when it was halfway down the stairs. Hilarity ensued.

[Retr0id]

Any chance you have a link to that comment?

[paganel]

Didn't bookmark it or anything and going back to the original reddit thread I now see that there are close to 9,000 comments, so unfortunately the answer is no...

[nrr]

Here you go. <https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/commen...>