[nickpsecurity]

We've been mitigating against this kind of thing with backups at other datacenters or colos for a while. They can be hot standby, cold standby, slightly degraded in performance, whatever. I also recommend the backup be on a different part of the overall power grid in case it cascades in failure. The good colo's often have connections to multiple backbones, too, which is extra redundancy.

That all assumes there's a total and catastrophic failure at main datacenter. If not, there's local backup batteries to sustain a smoother, fail-over plus shutdown. Plus, there's tricks like isolating the monitoring systems from main systems and power supply using things data diodes over octocouplers or infrared. At least one thing will still be working and feeding you reliable information over a wireless connection after the full failure.

NonStop and VMS setups from late 80's did better than Github. My own setups involving a minimum of servers plus apps with loose coupling could fail-over in such a situation. So, this just has to be bad architecture caused by who knows what. Examples below of OpenVMS in catastrophic situations having either no downtime or short downtime due to good architecture plus disaster planning.

Case study of active-active at World Trade Center http://h71000.www7.hp.com/openvms/brochures/commerzbank/comm...

Marketing piece where HP straight-up detonates a datacenter. Guess who was number 1 in recovery. :) https://youtu.be/bUwthF9x210?t=34s

[jacquesm]

I doubt it matters to anybody but was it really necessary to kill the fish?

[jakobegger]

Watch it until the end :)

[nickpsecurity]

Haha nice catch. I missed it originally thinking it would just be more marketing crap. So, they probably just moved it before detonating. Not sadistic bastards after all.

[jacquesm]

Watch 'The Prestige'.

[nickpsecurity]

I know... My original reply mentioned two scenarios with one having replacement fish. Then, I thought people would think I'm overly paranoid or negative. I just couldn't help wonder if they'd blow the fish for fun then avoid liability with similar looking ones. Then, I edited the comment for sake of presumption of innocence.

But, yeah, I hear you... Great movie as well. One of few that brings my favorite mad scientist into eye of mainstream audience as well. I doubt I must name him. :)

EDIT to add: I'm guessing you think the geeks were too sadistic to pass up the opportunity, eh?

[jacquesm]

> I'm guessing you think the geeks were too sadistic to pass up the opportunity, eh?

No, it just seems weird, that's all. I don't see how either interpretation would benefit HP.

[jacquesm]

I did, but unless the whole video was a fake it doesn't really matter does it? And if it is then that does not reflect well on HP either...

[nickpsecurity]

I didn't think about the image angle. Yeah, you'd think a marketing person would be like, "Wait, this could lead to a PETA lawsuit and lower sales. Not to mention our segment that likes fish."

[nickpsecurity]

Dude, I was thinking the same thing! That was seriously f*ed up. They should've left some cool fireworks or something left-over from July 4th. Or some safe-ish chemical that would make colorful smoke. All kinds of tricks you can do without killing live animals.

I mean, I've heard about things so wrong and ease it's like shooting fish in a bucket but... exploding fish in a datacenter? That's on another level.

[nacs]

> HP straight-up detonates a datacenter

Apparently 5 server racks in the middle of an open field is a "datacenter".

[nickpsecurity]

Nah, a collection of computers with high-availability setup communicating with another collection constantly over a dedicated or high-bandwidth line. In the demo, it was 5 racks in an open field. In the bank study, it was a whole bank's worth of computers in two locations. For some organizations, it's 5+ of them just to be sure.

The common trend is that the systems constantly sync critical data, can detect downtime, and automatically (or manually) fail-over when it occurs. Been OS's and ISV's offering that capability with many proven in field going back decades. Certain high-tech companies just don't apply those for whatever reasons. Maybe their stacks just still don't have that feature.