[Roark66]

Surprising that they don't have a geo-failover location.

When I worked in the DR field many years ago all but the smallest companies had some sort of business continuity plan if disaster strikes, but perhaps this is an example of positive bias as only companies that already cared a lot about DR would get a DR consultancy.

Those few companies I saw that had no DR and BCP whatsoever were usually small companies that grew recently and their insurance provider told them to make such plans so they have lower insurance premiums.

[chrismeller]

Having been through a variety of audits where this is invariably a topic that comes up at couple of different companies you also have to keep in mind that it’s one of those situations where one size doesn’t fit all.

A company with a call center had to have a plan in place for replacing office space (and, yikes, call center personnel) because having people available to answer calls and meet their SLA requirements was the biggest risk. The servers... meh. It wouldn’t be pretty, but grabbing a machine at the nearest Best Buy and picking up one of the semi-recent offsite backups from an external hard drive at the CEO’s house would get everybody through.

At a dev shop with a variety of state and federal government clients the office couldn’t matter less, what was important was geo redundancy, so we had racks in VA and in somewhere out west, I forget where. Another instance where everything probably could have been run (for a few days at least) off a couple off the shelf boxes stuck under someone’s desk, but it would have to be immediately, and no one lived anywhere to do that practically, so the second DC was invested in.

It’s all about identifying your specific risk and figuring out what to do to mitigate it to a reasonable degree without breaking the bank. Apparently Oreilly didn’t think geo redundancy for their main site was worth that trade off.

[dd82]

As a matter of fact, we're on GKE for most of our stuff, having moved out about 6 months ago. There are some remnants left, however, and that's what's causing the issue today.

The site is online, via learning.oreilly.com. The anonymous pages and login were having issues, because the login was tied to something (not familar with the exact details) in the DC offline.

So if you were already logged in, you won't notice much of anything (hopefully!). But if your session expired and you have to re-log in or were trying to see an preview of a book, that's where the problem existed.

[apple4ever]

Really surprising. Even if they self-host everything, you can still design a cloud based emergency spin up configuration.