[nyx_land]

this isn't even the first time something like this has happened. it's literally a running joke in programmer circles that AWS East going down will take down half the internet, and yet there's absolutely zero initiative being taken by anyone who makes these sorts of decisions to maybe not have every major service on the internet be put into the same handful of points of failure. nothing will change, no one will learn anything, and this will happen again.

[bilbo0s]

That’s very different though. That’s avoidable. We all can easily have our services running in different data centers around the world. Heck, the non-amateurs out there all have their services running in different Amazon data centers around the world. So you can get that even from a single provider. Hardware redundancy is just that cheap nowadays.

This CS thing, there’s no way around. You use it and they screw up, you get hit. Period. You don’t failover to another data center in Europe or Asia. You just go down.

Hardware, even cloud hardware, is rarely the issue. Probably especially cloud hardware is not an issue because failover is so inexpensive relative to software.

Software is a different issue entirely. How many of us will develop, shadow run, and maintain a parallel service written on a separate OS? My guess is “not many”. That’s the redundancy we’re talking about to avoid something like this. You’d have to be using a different OS and not using CS anywhere in that new software stack. (Though not using CS wouldn’t be much of a problem if the OS is different but I think you see what I mean.)

Amazon, implementing failover for your hardware is a few clicks. But if you want to implement an identical service with different software, you better have a spare dev team somewhere.

[karambahh]

AWS East going down will (and has) cause(d) disruption in other regions. Last time it happened (maybe like 18 months ago), you ran into billing and quota issues, if my memory serves.

AWS is, as any company, centralized in a way or another.

Want to be sure you won't be impacted by AWS East going down, even if you run in another region? Well, better be prepared to run (or have a DRP) on another cloud provider then...

The cost of running your workload on two different CSP is quite high, especially if your teams have been convinced to use AWS-specific technologies. You need to first get your software stack provider agnostic and then manage the two platform in sync from a technical and contract perspective, which is not always easy...

[bilbo0s]

You just made the single point of failure your software stack hardware abstraction layer. There’s a bug in it, you’re down. Everywhere. Not only that, but if there is CS in either your HAL, or your application you’re down. So to get the redundancy the original commenter was talking about, you need to develop 2 different HALs with 2 different applications all using a minimum of 2 different OS and language stacks.

Why multiply your problems? Use your cloud service provider only to access hardware and leave the rest of that alone. That way any cloud provider will due. Any region on any cloud provider will due. You could even just fallback to your own racks if you want. Point is, you only want the hardware.

Now to get that level of redundancy, you would still have to create 2 different implementations of your application on 2 different software and OS stacks. But the hardware layer is now able to run anywhere. Again, you can even have a self hosted rack in your dispatch stack.

So hardware redundancy is easy to do at the level the original commenter recommends . Software redundancy is incredibly difficult and expensive to do at the level the original commenter was talking about. Your idea to make a hardware/cloud abstraction layer only multiplies the number of software layers you would need multiple implementations of, shadow run and maintain to achieve the hypothetical level of redundancy.

[tormeh]

It's very avoidable. Just don't use shit software.