[cmiles8]

AWS’s US-East 1 continues to be the Achilles heel of the Internet.

And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

[dlenski]

The idea that AWS's services are fully regionalized or isolated has always been a myth.

All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions.

And IAM is not a wholly independent software stack: they rely on DynamoDB and a few other services, which in turn have a circular dependency on IAM.

During us-east-1 outages it's sometimes possible to continue using existing auth tokens or sessions in other regions, while not possible to grant new ones. When I worked there, I remember at least one case where my team's on-calls were advised not to close ssh sessions or AWS console browser tabs, for fear that we'd be locked out until the outage was over.

[Roark66]

Anyone who thinks one cloud provider will provide them full resilience is fooling themselves. You need multicloud for true high availability.

But then you want to use the same stack across providers and all the proprietary technologies (even hidden from you with things like terraform) are suddenly loosing their luster.

[hnlmorg]

I don’t think any actually believes that.

What people usually think is “resilience up to a reasonable level of risk and cost”.

Multi-cloud is simply isn’t cost beneficial for 99.9% of problems.

And for a lot of businesses who talk about risk, saying “we followed AWS best practices but AWS went down” is an acceptable answer to the question of liability.

If you are in a position where AWS going down is a reasonable risk, then you’re already in a specialised enough domain to have engineers who understand how to deliver HA across different vendors.

[Roark66]

Depends on the product. I'd never guarantee 3 nines (a bit under 9h of downtime per year) to my customers of any product with moderate complexity without multicloud(except for a cloud provider issues that affect entire region).

On the subject of cost. Multicloud doesn't have to be expensive, but it requires foresight during the initial design not to lock yourself up in proprietary tech.

Things such as AWS cognito, lambda elastic beanstalk, SQS can make it completely uneconomical to do multicloud.

But let's say you run roughly equivalent services (load balances, DNS, vms, containers for example under GKE, s3 compatible object storage, etc) it doesn't have to cost an arm and a leg.

[doublerabbit]

I jest, Anyone who thinks multicloud will provide them full resilience is fooling themselves. You need colocated hardware for true high availability.

[stephenr]

> And IAM is not a wholly independent software stack: they rely on DynamoDB and a few other services, which in turn have a circular dependency on IAM.

When you dogfood your own Rube Goldberg machine.

[zaphirplane]

We should let the IAM service team know if this glaring gap the hn thread figured out /s

I’m 99% ;) certain dependencies of foundational services are a well discussed topic

[myroon5]

> outside of China

[Nitpick] There are a few more AWS partitions like GovCloud:

https://jasonbutz.info/2023/07/aws-partitions/

[dlenski]

Yes, I'm certainly aware of the other partitions. That's why I said all the public cloud regions outside China.

Yeah, "govcloud" is technically available to the public, although there are other partitions reserved for government use that are not, and the naming is a big hairy mess. Many service teams don't have any US-citizens-in-the-USA working for them, and they cannot in any way adequately support these regions.

My on-call experience improved significantly when I moved from the US to Canada, and I got taken off the (extremely thin!) list of engineers eligible to ssh into RDS instances in Govcloud. There were so few USA-citizen-in-USA engineers that I had been getting tickets for services and instances in Govcloud about which I had only the very thinnest knowledge… and then I was limited in my ability to consult with others who were actually experts. The customers in Govcloud paid a premium to be there, I got paged for a bunch of tickets which I was ill-prepared to handle, and it was generally a bad experience for everyone.

Working with the airgapped secret/top-secret partitions was even worse. You would get paged incessantly and then someone who was cleared for access but knew almost nothing about the service in question would have to go to a SCIF in the DC area, and you would exchange screenshots and text instructions with a turnaround time of hours or days.

[electroly]

Since this article was written, AWS also added European Sovereign Cloud as a partition: aws-eusc.

[sidewndr46]

Isn't this kind of circular dependency what lead to extended downtime a while back?

[superjan]

It reminds me of facebook. Staff was locked out of the office due to the outage they were supposed to fix.

[martin8412]

Luckily the plasma torch and bolt cutter didn’t require logging in with Facebook.

[jethro_tell]

It's basically what leads to extended downtime almost every time. There are just some things in the stack that are still single points of failure, and when they fail it's a mess.

[dlenski]

Yes, I concur.

Sometimes the circular dependencies get almost cartoonishly silly.

Like, "One of the two guys who has the physical keys to the server cage in us-east-1 is on vacation. The other one can't get into his apartment because his smart lock runs into the AWS cloud. So he hires a locksmith, but the locksmith takes an extra two hours to do the job because his reference documents for this model of lock live on an S3 bucket."

I made that example up, but only barely.

[somat]

We had a pair of machines. And some bright spark set them up to mount each others NFS shares. after a power outage "Holy mother of chicken and egg NFS hangs batman"

That was a weird job, fun, it was a local machine room for a warehouse that originally held the IBM mainframe, it still held it's successor "the multiprise 3000" which has the claim to fame as being the smallest mainframe IBM ever sold. But now the room was also full of decades of artisanal crafted unix servers with pick databases. the pick dev team had done most the system architecture. The best way to understand it is that for them pick is the operating system, unix is a necessary annoyance they have to put up with only because nobody has made pick hardware for 20 years. and it was NFS mounts everywhere, somebody had figured out a trick where they could NFS mount a remote machine and have the local pick system reach in and scrounge through the remote systems data. But strictly read-only. pick got grumpy when writing to NFS not to say anything about how the other database would feel about having it's data being messed with. Thus the circular mount.

Still was not the worst thing I saw. I liked the one system with a SMB mount. "Why is this one SMB?" "Well pick complains when you try to write to a NFS mount, but it's NFS detection code does not trip on SMB mounts." ... Sighs "Um... I am no pick expert but you know why it does not like remote mounts right. SMB does not change that, Do you happen to get a lot of corrupt indexes on this machine?" "yes, how did you know"

[roryirvine]

Oh, yeah, re-exporting NFS mounts via SMB was very much a thing in the early 2000s - something to do with their different approaches to flock() vs fcntl() handling. If you ran into locking issues with nfs, then re-exporting via SMB was standard advice.

At some point, the behaviour changed and locks starting conflicting. IIRC, we hit it when upgrading to Debian Etch and took the time to unwind the system and make pure NFS work properly for us. Plenty of people took the opposite approach, and fiddled with the config to make locking a noop on SMB. I know of at least one web hosting company who ended up having to restore a year's worth of customer uploads from backups as a result...

[wgjordan]

A real example, from Facebook's 2021 outage [1]:

> Our primary and out-of-band network access was down, so we sent engineers onsite to the data centers to have them debug the issue and restart the systems. But this took time, because these facilities are designed with high levels of physical and system security in mind. They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them. So it took extra time to activate the secure access protocols needed to get people onsite and able to work on the servers. Only then could we confirm the issue and bring our backbone back online.

There was one (later denied) report that a 'guy with an angle grinder' was involved in gaining access to the server cage.

[1] https://news.ycombinator.com/item?id=28762611

[MichaelZuo]

Why would such a critical server even be accessible with only one set of keys?

I’ve always thought mission critical stuff needs two independent key holders, with key holes placed far apart enough to make it impossible for 1 person to reach both.

[bigfatkitten]

Other than for certain nuclear missile launches[1], that only happens in the movies.

[1] https://www.nationalmuseum.af.mil/Visit/Museum-Exhibits/Fact...

[michaelt]

They're not actually accessible with 'only one set of keys' in my experience.

You actually have to present your photo ID at the site entry gatehouse, then again to the building entry guard (who will also check you have a work permit and a site-specific safety induction) then you swipe a badge at a turnstile to get from reception into the stairwell, then swipe your badge at a door to get into the relevant floor, then swipe your badge and key in a code to enter the room with the cages then you use the key.

[sidewndr46]

a circular dependency and a single point of failure are not the same thing. If I have a single point of failure and it is down, I fix that and things work again. If I have circular dependency, there is no obvious way to fix anything that is broken any longer.

[grogenaut]

when you have a circular dependency, one strategy employed, is to have it be circular but interruptible for 18 or so hours. Call it an oh shit bar.

I'm glad I never had to get that deep into the failure chain.

[dlenski]

> Call it an oh shit bar.

Amazon's equivalent of this (sort of) was the "andon cord."

Not only was the physical metaphor that led to this name never properly explained (it's basically an "emergency stop" string in a Toyota factory), but the actual use of this mechanism was so heavily discouraged that I never saw it used in 4+ years at Amazon. except once very performatively by a VP who had already been paged awake at 2am or something like that.

In my experience, a lot of the AWS engineers live in continuous fear of screwing up by using the huge array of extremely powerful, dangerous, poorly-explained, and ever-changing tools that they have access to.

[zaphirplane]

Services outside of us-east-1 don’t call us-east-1 for IAM data plane thou right ?

[cmiles8]

They’re talking about the backbone and what goes on behind the scenes. There have been issues with services in other regions when us-east-1 has issues.

Folks built in other regions believing they were fully isolated only to discover later during an outage that they were not.

[master_crab]

IAM isn’t even really the most painful dependency. Route53 is. The control plane only runs out of use1.

Better make sure the only DNS operations you run during an outage are data plane queries and health check failovers.

[easton]

They actually kind of fixed this recently, you can ask them to move your route53 control plane to another region in the event of us-east-1 breaking: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/ac...

There’s a bunch of caveats but it’s worth enabling if you’re changing dns all the time (as most AWS networking doodads like to do).

[trollbridge]

Is there an architectural reason it’s not for replicas in the other AZs?

[jmsgwd]

> The idea that AWS's services are fully regionalized or isolated has always been a myth.

This is highly misleading. It's true that there's a handful of global AWS services - but only their control planes operate from a single region (e.g. us-east-1). Their data planes are regionally isolated or globally distributed.[1]

The only time you'd normally use a service control plane is to deploy changes, e.g. when you create, read, update or delete service resources or update configuration during a change window.

Workloads should be designed for "static stability", as recommended by AWS.[2] A statically stable workload only depends upon the data planes of the services it uses at runtime. Statically stable workloads are designed to continue operating as normal even if there's a service event impairing one or more control planes (including for global services).

> During us-east-1 outages it's sometimes possible to continue using existing auth tokens or sessions in other regions, while not possible to grant new ones.

This is just plain wrong! The IAM Security Token Service (STS), which grants IAM tokens, is a data plane-only service and runs independently in each region [3]. The IAM data plane, which enforces access control, is also regional.

If the IAM control plane is impaired, you might not be able to create new IAM roles (a control plane operation) - but you can continue generating and using temporary credentials for existing IAM roles (data plane operations) within the region your workload is running in. This allows statically stable workloads to continue using IAM without interruption.

[1] https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso...

"Global AWS services still follow the conventional AWS design pattern of separating the control plane and data plane in order to achieve static stability. The significant difference for most global services is that their control plane is hosted in a single AWS Region, while their data plane is globally distributed."

[2] https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso...

"...eliminating dependencies on control planes (the APIs that implement changes to resources) in your recovery path helps produce more resilient workloads."

[3] https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso...

"STS is a data plane-only service that is separate from IAM, and does not depend on the IAM control plane."

[dlenski]

You're right of course to distinguish the control plane and data plane, and it sounds like you know more about this than I do for IAM.

I disagree, though, that my post was "highly misleading" despite this omission.

As a practical matter, some services fail to achieve the "static stability" you describe, in terms of not depending on other services’ control planes.

And also, many on-calls ops and firefighting tasks (to say nothing of canaries and other automated tests) depend on other services’ control planes.

And above all, many AWS engineers (myself very much included even after years there) don't have a clear understanding of the boundaries of other services’ control planes. https://news.ycombinator.com/item?id=48078254

> > During us-east-1 outages it's sometimes possible to continue using existing auth tokens or sessions in other regions, while not possible to grant new ones.

> This is just plain wrong! The IAM Security Token Service (STS), which grants IAM tokens, is a data plane-only service and runs independently in each region.

I didn't mention STS in the service to which you're responding. The service that I worked on the most, RDS, required ssh'ing into live instances to solve basically all non-trivial problems (I'd guess 80% of the tickets that I saw actually resolved required it). And I have no idea if it how STS was involved in generating the ephemeral Midway-signed ssh keys required for it… but whenever there were us-east-1 IAM outages we'd have big problems opening new sessions, while less-capable web-console-based ops tools with long-lived credentials would keep working.

[Eridrus]

People say this, but this this was just a single AZ, and in the last 3 years of running my startup mostly out of use-1, and we've only had one regional outage, and even that was partial, with most instances uneffected.

And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol.

[skywhopper]

It wasn’t even all of a single AZ. None of my resources in use1-az4 had any issues. The most annoying thing was the 20 notifications we got saying “it’s not all fixed yet” every hour.

[linsomniac]

>And honestly, everybody else's stuff is in use-1

Yeah, but why put your eggs in that basket? I moved all our services from east to west/oregon a decade ago and haven't looked back.

[electroly]

Not OP, but I do single-region us-east-1 for a few reasons:

1. The severity and frequency of us-east-1 outages are vastly overstated. It's fine. These us-east-1 outages almost never affect us. This one didn't; not even our instances in the affected AZ. Only that recent IAM outage affected us a little bit, and it affected every other region, too, since IAM's control plane is centrally hosted in us-east-1. Everybody's uptime depends on us-east-1.

2. We're physically close to us-east-1 and have Direct Connect. We're 1 millisecond away from us-east-1. It would be silly to connect to us-east-1 and then take a latency hit and pay cross-region data transfer cost on all traffic to hop over to another region. That would only make sense if we were in both regions, and that is not worth the cost given #1. If we only have a single region, it has to be us-east-1.

3. us-east-1 gets new features first. New AWS features are relevant to us with shocking regularity, and we get it as soon as it's announced.

4. OP is right about the safety in numbers. Our service isn't life-or-death; nobody will die if we're down, so it's just a matter of whether they're upset. When there is a us-east-1 outage, it's headline news and I can link the news report to anyone who asks. That genuinely absolves us every time. When we're down, everybody else is down, too.

[coleca]

Sometimes you need capacity and you have to choose where the capacity is not where you would like it to be. Unfortunately, the days of cloud bursting, and thinking of the cloud as an unlimited resource where you can spin up and spin down machines at will is vanishing. Power availability and supply chain lead times combined with unprecedented demand are the reason for this. That's why you see all the hyperscalers recently reporting on their "backlog" in their earnings reports.

[christina97]

But it’s okay to be down when the whole internet is down.

[Eridrus]

90% of customers are located in use-1. Latency to use-1 is more important than being up when everyone else is down.

[nilamo]

> And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol.

Is it not a selling point to be able to say "we're still up while out competitors are down"?

[Eridrus]

We do win some deals on reliability, and have lost some, but AWS outages are not an important factor in any of this.

It's not that I think reliability doesn't matter at all, but most of our services are not multi region and it's bot the driving factor in SaaS reliability.

[bink]

It's worse when your region has issues and your customer's infrastructure is fine.

[skywhopper]

If you’re the one that’s down while no one else is, suddenly it becomes your fault.

[grogenaut]

none of my stuff is in us-east-1. I chose that specifically 15 years ago. Been a great decision.

[999900000999]

Too many people are using it.

In fantasy magic dream land loads are distributed evenly across different cloud providers.

A single point of failure doesn't exist.

It worked out with my first girlfriend. The twins are fluent in English and Korean. They know when deploying a large scale service to not only depends on AWS.

Healthcare in the US is affordable.

All types of magical stuff exist here.

But no. It's another day. AWS US-East 1 can take town most of the internet.

[afro88]

Core AWS services use it too. Even if you are hosted in another region, you can still be affected by a US-East 1 outage

[999900000999]

The idea would be to actually load distribute between different cloud providers.

But even then , the load balancer needs to run somewhere. Which becomes a new single point of failure.

I’m sure someone smarter than me has figured this out.

[jethro_tell]

yes, they have. It just costs a shit ton of money and is extremely difficult to get the suits to sign off on TWO full 'cloud services' bills. It generally doubles your cost and workload and increases your uptime by a couple hours/year, assuming you don't have bugs that affect one or the other cloud in your deployment stack.

It's basically a wash for almost all organizations for twice the cost and effort.

[999900000999]

Ok...

But where does the load balancer actually run. Does load balancer main run on AWS, and load balancer backup on Oracle?

[dboreham]

Short TTL DNS or BGP anycast.

[grogenaut]

also these things don't go down THAT often... well aws, not some others. More uptime that you probably had before. even the stock market takes a few days off every decade. Just ask W.

[justinclift]

> not some others.

Looking at Azure and GitHub in particular. ;)

[JackSlateur]

DNS

[erikerikson]

Not really. Your clients can random robin to connection points across providers and move write heads upon connection. If you worry about hard coding you can reduce the surface to a per-context first minimum contact point.

[leetrout]

Bingo. This is the one most people don't know about.

[b40d-48b2-979e]

I was surprised recently when setting up cloudfront with aws certs that it forced me to use us-east-1 to provision the certs.

[kbbgl87]

STS is only on us-east-1 I believe

[dlenski]

Yep. All of the identity and access management services for the non-China public cloud are in us-east-1. https://news.ycombinator.com/item?id=48071472

[avereveard]

All the control plane. Data plane is distributed and roles using iam to access resources can still do so during a control plane outage.

[dlenski]

Yes, you're right, but in my experience the boundary between the data plane and the control plane is not always clear, and especially unclear on these foundational and basic services.

There were enough "surprisingly control-plane" IAM operations in the AWS services that I dealt with, so we had to exercise extreme caution during outages.

[echelon_musk]

> It worked out with my first girlfriend. The twins are fluent in English and Korean.

You were dating twins as a form of redundancy?!

[dnnddidiej]

Dual writes. You'd need to have the same conversation with both to keep them in sync.

[keeganpoppen]

anecdotally (well, more "second-hand-ly i heard that..." it sounds like there were some carry-on effects on us-east-2 as a result of people migrating over from us-east-1, so, yeah... kinda hilarious how the multiple region / AZ thing is just so plainly a façade, but yet we all seem to just collectively believe in it as an article of faith in the Cloud Religion... or whatever...

[qaq]

It's no magic given the size of us-east-1 there is no spare capacity to absorb all the workloads

[8organicbits]

One of the SRE tricks is to reserve your capacity so when the cloud runs out of capacity you're still covered. It's expensive, but you don't want to get stuck without a server when the on-demand dries up.

[cherioo]

Is it really failing more, or we just don’t hear about failure happening elsewhere?

Last i heard azure outage it wasn’t even on HN frontpage

[stingraycharles]

It really is failing more, and it’s well known amongst industry experts. It’s the oldest, largest, and most utilized region of AWS.

I’ve heard people say that the underlying physical infrastructure is older, but I think that’s a bit of speculation, although reasonable. The current outage is attributed to a “thermal event”, which does indeed suggest underlying physical hardware.

It’s also the most complex region for AWS themselves, as it’s the “control pad” for many of their global services.

[adriand]

What kind of reputation does ca-central-1 have? I’ve been using it and it seems quietly excellent. Knock on wood.

[jedberg]

Most of the other regions are fairly stable. Ohio (us-east-2) is a great choice if you're just starting out. Not sure about ca-central-1, but I've never heard anything bad about it.

[dlenski]

It wasn't heavily utilized when I worked at AWS, until 2024.

If your customers are clusterrd in Toronto and Montreal, it probably makes a lot of sense to use ca-central-1. If you've got a lot of customers in Western Canada, us-west-2 is gonna have better network latency.

Other than a couple regions that had problems with their local network infrastructure (sa-east-1 was like that), there's little or nothing to differentiate the regions in terms of physical infrastructure and architecture.

[dehrmann]

> building across multiple regions and AZs is a thing

If you do this for resiliency, be prepared to pay the capacity tax (2 regions means 2x capacity, 3 regions means 1.5x), have the machines already running in a multi-region setup (don't expect to be able to spin up instances or even get capacity during an outage), and ready to deal with the added complexity of multi-region hosting.

[coredog64]

There’s all kinds of fun pitfalls with multi-AZ. Like you can create RDS subnets across multiple AZs but then you can’t remove an AZ. Which really sucks when your core database covers all 5 us-east-1 AZs and randomly can’t failover because you picked an instance type that use1-az4 can’t host.

[ohnei]

I've always been impressed by Amazon's ability to present the shittiest experience possible and imply the blame is with things like isolation that they don't really provide.

[y3ahd0g]

No. This is nonsense.

Some SaaS apps had issues.

The Internet was fine.

This is physical reality. The internet was designed to route around this.

Just because some app devs do a lazy job doesn't mean the entire infrastructure as designed is garbage.

Just because some app devs are over reliant on a single cloud service doesn't mean the Internet is broken.