[mmalone]

This bug affected several "supported" AMIs running 2.6.32 series kernels that we tested at SimpleGeo, including the official AMI released by Canonical. After we ran out of patience debugging this stuff we contacted Amazon and worked on the issue with a guy from their kernel team (who was really helpful, fwiw). He agreed that the behavior was bizarre and opened an upstream bug with Canonical [1].

You're sort of contradicting yourself here. You suggest that the distro you're running is independent of the kernel version you're running. But then you go on to claim that this bug was introduced by someone who was not running the default supported kernel. Are you saying that people should run the supported kernel, and be tied to whatever's supported upstream, or are you saying they should risk building their own? Clearly there are benefits and drawbacks either way.

[1] https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/708...

[moe]

No, I'm saying you should run the kernels that Amazon provides and/or a time-tested one.

[mmalone]

Amazon's official linux build, at this time, is a custom distribution that uses RPM and is different from the Ubuntu/Debian world in several significant ways (e.g., a new libc implementation). Migrating your WordPress blog to a new platform might be easy, but when you're managing hundreds of machines running thousands of packages that sort of change is not trivial.

Running "time-tested" kernels is not really the best advise either in this case. Xen is a fairly new environment, and EC2's implementation has some quirks, so there's a pretty regular stream of bug fixes and other improvements in recent kernels that are often worth picking up. If I went to Canonical with a "time-tested" kernel bug they'd tell me to upgrade before they'd give any real support.

When we talked to Amazon about switching to their AMIs they advised us that it was probably _not_ worth switching, that switching might not fix the problem, and that the AMIs we were running were widely used and supported. They made it clear that they work closely with Canonical and other providers to get high quality AMIs into their ecosystem. Long story short, the people who you admit know the most about the EC2 environment advised us that they weren't necessarily the best option, or at least not the only option, for good AMIs (sort of like how hardware manufacturers aren't the best option for an operating system).

So the answers aren't really cut-and-dry here. Every time Amazon changes their dom0 there's a chance your "time-tested" kernel will stop working. And just because Amazon runs the infrastructure doesn't mean they're the best choice for a Linux distribution.

[moe]

You keep lumping the linux distribution in with the linux kernel. They are separate things. You can run Ubuntu on an Amazon AKI. I run Debian on Amazon AKIs. And if the latest Ubuntu depends on a particular kernel feature that the tested kernels don't have then it's probably not a good idea to run the latest Ubuntu.