[implying]

This is a bizarre analysis. Public legal risk is absolutely the last imaginable threat to us-east-1, short of aliens abducting it. The U.S. security apparatus depends on AWS and would never allow it, Wall Street would never allow it, never mind the fact that Amazon itself would leverage every tool at it's disposal to protect its reputation for reliability. The politicians involved in this scenario might seek to remove Amazon's competitive advantages, or fine them, but the people who understand what AWS even is would never consider a move to shut down a datacenter.

Both the "enemy action" and "operation failure" scenarios are much bigger risks than this article makes out to be. Every non-aligned nation-state offensive cyber team has a knockout of us-east-1 at the top of their desired capabilities. I'm sure efforts range from recruiting Amazon employees to preparing physical sabotage to hoarding 0days in the infrastructure. There's no reason to think one of them wouldnt rock the boat if geopolitics dictated.

Operational failure is probably the most likely. AWS might have a decade of experience building resilience, but some events happen on longer timescales. A bug that silently corrupts data before checksums and duplication and doesn't get noticed until almost every customer is borked, a vendor gives bad ECC ram that fails after 6 months in the field and is already deployed to 10,000 servers, etc. Networking is hard and an extended outage on the order of a week isn't completely impossible. How many customer systems can survive a week of downtime? How many customer businesses can?

[dijit]

> Amazon itself would leverage every tool at it's disposal to protect its reputation for reliability.

This is a joke, right? The _real_ degradation map of us-east-1 of the last 5 years looks significantly worse than my non-UPS backed Home PC in Sweden.

Personally I'm not looking at us-east-1 as reliable at all; they even suffered a "harddrive crash" https://www.bleepingcomputer.com/news/technology/amazon-aws-...

[michaelt]

I assume the author isn't asking "What if there was a 30 minute outage of us-east-1" (just wait for it to come back) or "What if there was an outage of a single AZ in us-east-1" (just spin things up in a different AZ)

Rather, they're asking "What if there was a 30 day outage of us-east-1" - so anyone who isn't multi-region or multi-cloud loses everything, including backups, AMIs, and control plane access.

(FWIW I agree with people disagreeing with the worry levels in the article - a solar storm last seen in 1859 is more likely than a software bug? Ha!)

[jdub]

Please don't mischaracterise the point: The solar storm is not more likely than a software bug. The effect of the solar storm is more likely than a similar effect by a software bug (particularly in the context of AWS).

[dijit]

realistically if AWS is down for 1+ day in that region you will be wondering very much if they're coming back in just 1hr more, 1 more day, 1 month or if at all.

Yeah, it's kind of "incredible" that they'd be gone for so long, but I've been on AWS during long outages before and it's never certain how long it will go on for and if data will be kept.

We assume too much I think of our providers, and I believe that's what the post is about. If you already _know_ AWS will be down for 30 days then you can make an informed decision based on that.

For me, there's no difference between 8 hours or 80, if you're down more than 8hrs I'm going to activate my redundancies. But: I have redundancies. Many people don't.

[ninju]

Curious has AWS us-east-1 ever been down for more that 8+ hours at a time?

[kevingadd]

"It hasn't happened since 1859" has nothing to do with whether it's likely to happen soon. You can get heads 162 times in a row when flipping a coin, it's not likely to happen but it's possible. In this case the chance of it happening is not 50%, so us going ~162 years since the last one isn't that weird. That doesn't mean the chance is 0% - we could lose that coin flip this year.

This is related to Tim's mention of climate change - lately, '100-year' and '10-year' storms are happening more often. It's possible the changes to our climate have turned those 100-year storms into 10-year storms, but we may not know until the observations and climate models catch up.

[skywhopper]

The solar storm would be more likely to, as you said, take down the region for a significant period of time. A software bug is more likely to happen, but less likely to have that big of an impact.

[klik99]

Surprisingly, this was published (though maybe not written) just after FB locked out all their employees with a bad config...

[praptak]

Yeah, the relative probability of scenarios may be wrong. Maybe even some of them are totally bogus. This doesn't undermine the main point of the article though.

I think the main point is that there's quite a lot of eggs in that basket and we should see this as a problem. Any single organization can think they have contingency plans for a big cloud region going permanently down. The problem is that when all of them try to execute their plans at once it won't work.

[deepstack]

While your analysis is sound, I wouldn't disregard a Carrington type of event. It can happen again and we never know where. None of our current electronic infrastructures are hardened to handle this kind solar storm/EMP. IPFS is a good direction in mitigate these kinds of centralised data risk.

Diversifying one's cloud/server provider is a good thing! Or simply don't rely only on the cloud. Storage devices are cheap now days, just have local backup and/or in different geographical location.

[fnord123]

It needs to be put in context. If a Carrington type event takes place then much of the tech we take for granted will be offline. If you do GPS navigation it doesn't matter if route planning is offline because the satellites themselves might be broken. It doesn't matter if you sell makeup because the delivery drivers, planes, and boats will be unable to navigate adequately to delivery anything. If doesn't matter if you stream videogames because many ISPs may well be offline.

[tuyiown]

Off-topic (and I'm totally not expert, please please correct me if i'm woring) but the the real risk of a Carrington type of event is for the power grid, especially the destruction of equipment at end of power lines.

The idea is that those large transformer cannot be mass produced and could be completely destroyed.

The cool hack is that physical disconnection in time avoids those damages.

At small scale the difference of potential is not enough to fear much physical damage.

[VHRanger]

Uh IPFS doesn't have replication? It's garbage in terms of reliability as far as I see it

Even 20 year old BitTorrent is a better option if that's the risk you're considering

[MattGaiser]

> Bear in mind that Republicans hate Amazon because of Bezos’s Washington Post and because the whole tech industry is (somewhat correctly) perceived as progressive

Do Republicans actually hate Amazon all that much or do they just go on Fox News or Twitter and proclaim that they do? As much as they might complain about certain corporations, they don't seem to be at the top of the hit list.

[listless]

I don’t think they hate amazon, but they certainly don’t appreciate WaPo or NYT or fill-in-the-blank left leaning media (read most). How that translates to shutting down an Amazon data center just kind of shows how deranged people get when it comes to politics.

[javajosh]

Trump seems to at once despise Bezos for bailing out the WP, but also admires and fear him, since Bezos' money makes Trump look poor in comparison, and the working conditions in Amazon warehouses and anti-union activity reflects the kind of autocratic values that Trump admires. It's pretty clear that Trump would love to be Jeff's friend, which gives Amazon even more leverage to survive any right-wing populist attack.

[paul_f]

This section destroyed the authors credibility altogether in my mind. Seems the entire article was a set up to make a political attack.

[condiment]

The best case scenario for an operational failure is the loss of a single AZ, but many of the scenarios you described are things that could impact an entire cloud vendor simultaneously. As others have pointed out (in discussing the Carrington event and the recent Facebook outage), it's not a matter of if it will happen, but a matter of when. And then it's just a question of duration and scope of the impact.

At this point around half of the world's leasable compute is concentrated in fewer than 100 facilities, the locations of which can easily be found with a google search. Using public satellite imagery you can identify network connection points as well as follow power transmission lines. In a wartime scenario, these are industrial targets with astounding strategic value, a tiny geographic footprint, and limited collateral damage in terms of human life. The Nagasaki and Hiroshima of the future could simply be kinetic attacks against a couple of datacenters. I'm alarmed that nobody is prepared for this and the industry zeitgeist seems to be to continue the consolidation of our economies into the cloud.

[godtoldmetodoit]

I agree datacenters would clearly be targets, but our IT infrastructure is hardly alone in being a relatively concentrated strategic target.

If kinetics are in play, said actor could also destroy our oil refining and pipeline systems. Taking out a few dozen large baseload power generation facilities would have a massive impact on the grid.

[chihuahua]

Regarding "enemy action" - I wonder what percentage of AWS/GCP/Azure employees are also employed by foreign security services? Is it 0%, 1%, 5% ?

[snotrockets]

You have to assume it's >0%, for both foreign and domestic, and I'm certain the cloud providers assume that too.

[lenkite]

I would love to see Wall Street prevent a Coronal Mass Ejection.