[beneth]

We are running our sites across three AZs, all in US-EAST. Our three web servers (one in each AZ) are not even able to load static test pages from their EBS volumes at this time. We also have a Percona MySQL cluster across three AZs and we are seeing very degraded perfomance in the EBS volumes on all three servers as well. It seems like this is impacting more than a single AZ from what I'm seeing.

[blantonl]

Well, we've gone from "degraded performance" to completely losing our EBS instance store for our master MySQL server. If we don't see resolution shortly we might have to look at promoting one of our slaves and moving forward from there.

[blantonl]

We ended up promoting a slave to master, and moving forward from there. Our master eventually returned to service, and we'll promote him tomorrow after things settle down.

Interesting to note that we issued a reboot on our Master and it went away and didn't return for over 45 minutes - we thought for sure it would have to be terminated. API calls and console access was severely restricted, so even launching in new AZ's was problematic.