[x0x0]

Netflix is a classic case of very bursty demand when they eg regularly re-encode all video, or even just see huge audience swings when people get home from work and turn the tv on.

Most companies at even 1k servers will get enormous cost savings from O&O.

A former employer, then in the 600-ish box range, costed out moving to ec2. Our ops actually brought up our internal hadoop cluster on ec2 as well as ran parallel workloads for a week. Amazon at the time offered price deals to attempt to get them as a marquee customer, as well as a bunch of engineering time. Even so, ec2 cost 3-4x as much for 1/3 the throughput. ec2's prices have since fallen, but so has the price of bare hardware.

People also overestimate the cost of hands-on; this company contracted for 4 hours / week with a local person. If you get serious about puppet/chef you can successfully run lots of servers in a remote datacenter.