[mihasya]

You're neglecting the fact that which distro you choose has a large influence on the kernel version you get to run. With the amount of work going into stabilizing the kernel when running in a virtualized environment, chaining yourself to a slow-moving distro will cause the exact opposite of stability when running on Xen. See: this blog post.

[jerf]

Recompiling a Linux kernel with a custom patch is in the core set of skills that I would expect any Linux admin to have. It's not that hard. It's not even that hard to take existing distro kernels and add a patch to them, and maintain your patch going forward as the distro continues their refinements.

When you have Linux on your server, you own it. You can do whatever you want. The distro is the beginning of your power, not the end of it. If you're running a Linux server and you currently have the attitude that you are boxed in by your distro I recommend that you immediately dig into the relevant packaging system and learn enough to put your own patch on top of any existing software package, and recreate the package in the relevant manner (new RPMs, new .deb, whatever).

(Yes, there's a cost/benefit tradeoff to each such patch you have to carry on, but there is still economic value merely in having the option.)

[nupark2]

It's 2011, and screwing around with kernel compilation is a waste of my time. It should 'just work'.

The only time I recompile a kernel is when I'm working on kernel code. If UNIX distributions are doing their jobs, a sysadmin should never have to touch it.

[jerf]

It's 2011. Everything should just work.

Alas.

[nowarninglabel]

I really have to agree with the other reply that fooling around with re-compiling a kernel is definitely beyond what I would consider to be a rational expectation. You may not think it's that hard, and it may not be for someone whose sole job is to do just that, but for the vast majority of admins that are wearing many hats this is going to be a unneeded time sink.

[moe]

You're neglecting the fact that which distro you choose has a large influence on the kernel version you get to run

That's nonsense. Most EC2 AMIs are linked to the amazon AKIs which are unrelated to whatever distro the AMI contains. Most of my debian instances run on a kernel tagged "fc8xen".

The ability to chainload a self-compiled kernel on EC2 is a relatively recent invention (mid-2010) and I have yet to see a good reason to do that for linux.

The article does unfortunately not mention which AKI(s) are affected, but it seems likely this bug was introduced because someone figured "newer is better" and went with the latest Ubuntu kernel instead of sticking to a proven amazon AKI.

[mmalone]

This bug affected several "supported" AMIs running 2.6.32 series kernels that we tested at SimpleGeo, including the official AMI released by Canonical. After we ran out of patience debugging this stuff we contacted Amazon and worked on the issue with a guy from their kernel team (who was really helpful, fwiw). He agreed that the behavior was bizarre and opened an upstream bug with Canonical [1].

You're sort of contradicting yourself here. You suggest that the distro you're running is independent of the kernel version you're running. But then you go on to claim that this bug was introduced by someone who was not running the default supported kernel. Are you saying that people should run the supported kernel, and be tied to whatever's supported upstream, or are you saying they should risk building their own? Clearly there are benefits and drawbacks either way.

[1] https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/708...

[moe]

No, I'm saying you should run the kernels that Amazon provides and/or a time-tested one.

[mmalone]

Amazon's official linux build, at this time, is a custom distribution that uses RPM and is different from the Ubuntu/Debian world in several significant ways (e.g., a new libc implementation). Migrating your WordPress blog to a new platform might be easy, but when you're managing hundreds of machines running thousands of packages that sort of change is not trivial.

Running "time-tested" kernels is not really the best advise either in this case. Xen is a fairly new environment, and EC2's implementation has some quirks, so there's a pretty regular stream of bug fixes and other improvements in recent kernels that are often worth picking up. If I went to Canonical with a "time-tested" kernel bug they'd tell me to upgrade before they'd give any real support.

When we talked to Amazon about switching to their AMIs they advised us that it was probably _not_ worth switching, that switching might not fix the problem, and that the AMIs we were running were widely used and supported. They made it clear that they work closely with Canonical and other providers to get high quality AMIs into their ecosystem. Long story short, the people who you admit know the most about the EC2 environment advised us that they weren't necessarily the best option, or at least not the only option, for good AMIs (sort of like how hardware manufacturers aren't the best option for an operating system).

So the answers aren't really cut-and-dry here. Every time Amazon changes their dom0 there's a chance your "time-tested" kernel will stop working. And just because Amazon runs the infrastructure doesn't mean they're the best choice for a Linux distribution.

[moe]

You keep lumping the linux distribution in with the linux kernel. They are separate things. You can run Ubuntu on an Amazon AKI. I run Debian on Amazon AKIs. And if the latest Ubuntu depends on a particular kernel feature that the tested kernels don't have then it's probably not a good idea to run the latest Ubuntu.

[josephruscio]

Canonical maintains and updates their own AKIs for their official Ubuntu AMIs, which is what we're running in production. I suggest deferring to their judgement. Here's a full release history of the Ubuntu 10.04 server AMI's, note the changing AKIs and note that none of them are the default ones provided by Amazon:

http://uec-images.ubuntu.com/query/lucid/server/released.txt

[moe]

And their rationale for doing that is?

Looks like their judgement didn't work out so well this time. I'd be wary of running the latest untested kernel for no reason other than "because we can".

[mihasya]

So Canonical, a distribution maker, is not to be trusted for their kernel suggestions, but Amazon is infallible? On what basis?

We asked the Amazon kernel team if we should try switching to one of their kernels/distros, and they said "No, just upgrade to Maverick and the accompanying kernel." It's been pointed out that Maverick has its own set of Xen bugs. I guess Amazon doesn't know everything.

The horse you're getting on about using the "proven" Amazon kernels is a bit high. Turns out this whole virtualization thing is somewhat new, and the kinks are still being worked out. Old kernel builds don't work particularly well because a lot of their assumptions are broken by virtualization; new kernels are what they are - new.

(Edit, forgot initially): Finally, we ran 10.04 - the Long Term Support release of Ubuntu from a year ago. There was no "because we can."

Frankly, I'm a bit amazed at your disdain for people sharing their findings from practical experience running into these issues in high-load production environments.

[moe]

So Canonical, a distribution maker, is not to be trusted for their kernel suggestions, but Amazon is infallible?

Neither is infallible. But Amazon probably knows the intricacies of their platform better than Canonical. And they likely run some of their own stuff on these kernels for a while before releasing them to the public.

Old kernel builds don't work particularly well

Don't work as in what? This is the first time I hear about a kernel problem on EC2.

disdain

I don't see where I voiced disdain. I merely responded to the guy who claimed your EC2 kernel is linked to the distro you run. That's simply not true.

[mmalone]

If this is the first time you've heard about a kernel problem on EC2 you're probably not managing a very large EC2 infrastructure [1, 2]. Even in non-virtualized environments, at scale, it's common to run into linux kernel bugs, or at least peculiarities. Which is why large tech organizations invariably employ kernel dev teams.

The guy who claimed EC2 kernels are linked to the distro you run was simply claiming that, unless you want to go it on your own, you're tied to the kernel provided by a supported AMI. As you've suggested multiple times, there are benefits to running an environment that is supported and that other people have operational experience with. Honestly, I'm not even sure what you're arguing anymore... seems like you're just being antagonistic.

[1] https://bugs.launchpad.net/ubuntu/+source/linux-ec2/ [2] https://bugs.launchpad.net/ubuntu/+source/linux/+bugs?field....