[nzoschke]

Great article. The Sysdig team really knows how to root cause tough problems. The Sysdig tools can be invaluable for getting and making sense of low level data.

If you want to play with ELBs, rolling deploys, connection draining to ECS containers, I humbly submit the open source Convox project I am working on.

https://github.com/convox/rack

It sets up a peer reviewed, production tested batteries-included VPC, ECS, ASG, ELB, etc cluster in minutes.

If the conclusion of this Sysdig post was that you always need to run 2 instances per AZ for the best reliability, I would strongly consider adding that knowledge into the tools either as a default or a production check.

Since it sounds like an ELB bug I'll keep the 3 instances in 3 AZs default.

[kirbypineapple]

Do you have any thoughts on how to scale load balancers horizontally and on demand? I've played briefly with attempting some dynamic DNS routing based on health checks to re-route traffic from balancers that have been shut down due to low traffic, but DNS really isn't designed to work this way.

[tedmiston]

I'm not clear what you're asking... Do you mean auto scaling the EC2 instances in the load balancer? Or auto scaling the # of load balancers? Or something else?

Of course the former is very common with Auto Scaling Groups [1] [2]. Then you can use round robin or session sticky routing algorithms in the load balancers.

(Apologies if I'm totally off-base for what you were asking.)

1: http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide...

2: http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide...

[kirbypineapple]

I meant the latter; if your load balancers are overwhelmed how do you scale them? Further to that point, is it possible to create an architecture where load balancers are responsive and can spin up in response to traffic? If you have to deal with loads that are prone to bursts, you need to allocate those load balancers in such a way that they can handle the worst case scenario.

[tedmiston]

I haven't worked at the scale where dynamically scaling the load balancers themselves is the bottleneck. I think you pose an interesting question, and I'm hoping someone with more knowledge can comment on that.

[sphinx501]

I don't believe there is any way to scale out the ELB from user side; you can contact AWS support to 'pre warm' ELB's for high traffic sites before cutting over DNS to them.

http://aws.amazon.com/articles/1636185810492479#pre-warming

[jasonwatkinspdx]

DNS based load balancing has a lot of caveats due to how clients can hold onto stale information. To do better means stepping up to running anycast via BGP, a signifigant jump in complexity. That's why services like Cloudflare exist.