[dups]

Similar experience here. I have a few particularly memorable experiences of dealing with the fallout from the September reboots. Although to be truthful this is partly due to the fact we were moving office and I had people jubilantly packing up around me as I worked to keep things afloat.

Not wanting a repeat of this we have migrated as many services we can into autoscaling groups, and automated all resource creation with CloudFormation.

This was inspired by this excellent Netflix blogpost: http://techblog.netflix.com/2014/10/a-state-of-xen-chaos-mon...