[Sanddancer]

The reboots were because Amazon cuts a lot of corners in their Xen setup because instances are supposed to be "disposable". A proper VM cluster would use dedicated storage nodes that export over something like iscsi, which would require just transferring a memory snapshot, or would use the native, slower, disk snapshotting migration. But that's just one of many ways AWS is rather broken from an operations standpoint.

[oblio]

Thing is, the AWS philosophy of considering everything short of data sources "disposable" leads to a lot more robust engineering.

Yes, it makes Ops work more difficult, but as someone smart said regarding software: if something hurts, you're not going it often enough :)

[yellowapple]

I'm pretty sure "if something hurts, you're not doing it often enough" doesn't apply to, say, arm-breakage or self-immolation :)

[ownagefool]

What you call a proper VM cluster has been pretty terrible from my experience as both a developer on an early cloud and a consumer of large government facing clouds.

Typically the network or SAN becomes oversaturated and the vm's shit the bed. AWS on the other hand was considerably more reliable and I'd argue they've made better decisions rather than cut corners.

[brazzledazzle]

The SAN becoming over-saturated isn't something that just "happens". Between establishing limits ahead of time and monitoring that shouldn't happen without someone knowing well ahead of time.

[ownagefool]

Just to be clear, I'm defending Amazon over an accusation that they've cut corners from an install that was up in 2007 or earlier. Now we could have focused on xen guests shitting the bed for no apparent reason or flakey switch port but we decided to focus on storage.

OK so I'm not proclaiming to be an expert but as someone working in the area in 2007, buying something off the shelf like it aint no thing, you're getting something like a netapp with a limit of 512 iscsi inititatiors, or a Sun amber road where your only form of automation is an ssh consol with a big warning stating it's unsupported.

From memory, there was no such thing as setting quotas on the amount of IOPs an iSCSI inititatior can do, in fact, I'm fairly sure IOPs quotas just didn't exist period, as the vendors weren't really up-to-speed with this new selling vm's thing. So basically, we're suggesting that it's a good idea to just buy a SAN to run an indetermined amount of vm's that are going to do an indetermined amount of IOPs.

OK, cool, you're now indebted to storage vendors selling you new shelvs at £80,000 a pop for those extra IOPs you so deperately need. Now to be fair, Amazon could probably afford it, but your VMs would still be a lot more expensive and would probably have still been totally disposible when your switch port decides to blip traffic to the SAN or as previously stated, Xen shits the bed.

None of these things might be a problem today, I don't know, I'm more a consumer than a producer of clouds these days, but I'd suggests these criticisms are bullshit. They come from some obviously smart people, but bullshit none-the-less.