[lovedswain]

The global outage drinking game:

bad DNS config push - 1 shot

routing loop - 2 shots

third party advertising your routes - 3 shots

power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots

Wolves ate through fiber - 5 shots

And it was a full moon - 6 shots

Single service failure, but service has not been restarted in 5 years, and no longer restarts in any documented fashion - 7 shots

And service developers left the company to found a startup - 8 shots

Expired internal SSL certificate - 9 shots

Daylight savings changeover-induced database corruption - 10 shots

Windows Update - 11 shots

[erik_seaberg]

5 is often broken down to “rain fade” and “backhoe fade.”

My favorite can’t-restart story is “the shared password manager is down, we need a hardware crypto key to restart it, the key is in the safe, and the safe’s combination was in the shared password manager.”

[spondyl]

I was going to link a related comment but you're actually the author of it!

Here it is anyway: https://news.ycombinator.com/item?id=22316230

[reaperducer]

Coincidentally, I'm in the same predicament with Facebook, itself.

I can't log in to Facebook because Facebook wants approval from the Duo Mobile 2FA app. The 2FA app won't approve the login because it wants a code that can only be obtained by logging in to Facebook.

Solution: Deleted both Duo Mobile and Facebook apps. Moved on with my life.

[dylan604]

isn't that like throwin the baby out with the bath water? i've never had problems with duo mobile, but then i don't use FB so i've never seen how it behaves with FB. so dump FB, but why dump duo mobile?

[reaperducer]

Because the only thing I used it for was to login to Facebook. All of my other 2FA is through other methods.

[lovedswain]

In a similar vein, the alternative option for 7 was a cyclical dependency introduced since last restart

[erik_seaberg]

I heard a story that Google SRE had to up a sacrificial Bigtable in any new zone before they could up enough services to finally up a real Bigtable for the zone.

[neura]

Please, please tell me this is a true story. It's amazing.

[erik_seaberg]

I can only verify being unable to get the wi-fi password. Otherwise it’s all secondhand, I wasn’t on call.

[ASalazarMX]

> Wolves ate through fiber - 5 shots

I hope it's wolves, that would create a new category in https://cybersquirrel1.com/

[partcyborg]

You forgot:

Hunter shot your fiber relay with a hunting rifle - 6 shots

[michaelt]

6 shots? It only needed one.

[neura]

depends on the gun... some people have interesting ideas on hunting.

[godelmachine]

DST can be such a hassle. It affects our reports and change requests.

[erik_seaberg]

An adtech company once freaked out over a sudden 4% revenue drop, until someone pointed out that yesterday was 4% shorter than the day before.

[alfalfasprout]

Don't forget cosmic rays!

[nabla9]

Apple decided to shut down Facebook.