[jespern]

It was fixed around 4am (GMT+2) last night, with the assistance of Amazon. I'm just going to summarize what happened here:

We were attacked. Massive UDP DDOS. The flood of traffic prevented us from accessing our EBS store with any acceptable speeds, which is what caused everyone to think the problem was between our EC2 and the EBS. Of course this also explains why booting up a new instance and EBS didn't help anything.

Also, it's happening again now, and we're working with Amazon to remedy it once more.

[tlrobinson]

Is there anything Amazon could have done to prevent this (or at least made diagnosing it easier), or is it a problem with your particular application?

[jespern]

We're talking UDP flood here, saturating our bandwidth. It never reached our servers, it just ate all the bandwidth on our connection. I guess what Amazon could have done is be quicker in spotting the DDOS and take measures to prevent it.

[spudlyo]

So you never saw any evidence of this DDOS yourself? I'm somewhat skeptical of this explanation. It seems to me with shared infrastructure it'd be difficult to saturate just one customer's connection. It also doesn't make sense to me that this could be done without the traffic ever reaching your server. You used the phrases "our bandwidth" and "our connection" do things really work this way on the AWS cloud?

Anyway, I'm really sorry you guys had to go through all of this, and I hope whatever it is that caused it is fixed.

[tlrobinson]

So it was actually entirely unrelated to EBS? The reason it was taking 10 seconds to do an "ls" was simply a saturated connection to your server, not too much EBS activity?