[SpikeGronim]

I'll share how wavii does this. We use amazon elb and chef. The first chef recipe that runs on our frontend is "touch /tmp/website_should_drain". The website then returns 404 from the /status health check URL. We wait 30s, update the site, and then rm the draining sentinel file. If the deployment fails the host stays out of the elb. If more than 1/3 host fail we abort the whole deployment. This works well for us and was very simple to implement.

[zwily]

We've found that when the ELB health check returns non-ok, the ELB will immediately kill any requests in flight to that box. So it's kind of impossible to do a real "drain" with ELB. Are you seeing different behavior?

[mattyb]

We've seen the same behavior, so we do the draining via HAProxy, which does the right thing.