Has anyone actually done this at scale and seen how it works? We've talked about it a few times because Consul looks really cool but it doesn't appear to have the full capabilities of SmartStack yet, in particular Synapse (https://github.com/airbnb/synapse) for intelligently doing stats socket updates, intelligent reloads, or controlling how HAProxy reloads over time.
Also consul only has two levels of services right, "local" and "remote"? I'm not sure that is sufficiently powerful to describe SOA's which need to be able to differentiate services on the machine, rack, colo, datacenter, region locality? I suppose you could do what we do in SmartStack and just register the same service instance at a few different places since I think that consul can register arbitrary services.
Maybe all we need is a consul watcher in Synapse, and a consul-template for the Synapse configs, and we can get the best of all worlds. With Linux 4.4 SYN handling in Linux might even be fixed by Eric Dumazet being super hardcore (e.g. https://lwn.net/Articles/659199/) and we won't even have to worry about reloading HAProxy.
Well it depends on how you implement it. I wouldn't imagine that each microservice has it's own haproxy-consul-template router. If it did, then sure, you're going to swamp it.
Maybe I'm not understanding the situation correctly, and granted, I've not ran it on a cluster of more than 5 machines with more than 10 microservices as we decided to use Rancher instead.