[mlyle]

> since engineers got locked out of their systems

Sounds like the same happened here:

"Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. We have backup procedures for handling such an event and used them to take control of the affected locations."

But Cloudflare had sufficient backup connectivity to fix it. I'm curious how Cloudflare does that today-- the solution long ago was always a modem on an auxiliary port.

[Melatonic]

Worst case if I was designing this I would probably have a satellite connection running over Iridium at each of their biggest DC's

Also lets face it - the utility of a trusted security guard/staff with an old fashioned physical key is pretty hard to screw up!

[nijave]

Not sure how common it is, but you can get serial OOBM devices accessible over cellular which would then give you access to your equipment.

I'm surprised more places don't implement a "click here to confirm changes or it'll be rolled back in 5 minutes" like all those monitor settings dialogues

[cwt137]

They have their machines also connected to another AS, so when their network doesn't/can't route, they can still get to their machines to fix stuff.

[jve]

> the solution long ago was always a modem on an auxiliary port

Now you can use mobile Internet (4G/5G)

[ccakes]

Cell coverage inside datacenters isn't always suitable, occasionally even by-design.