[jolynch]

Funny enough we're doing something pretty similar for our external lb tier but instead of a custom proxy we're just using nginx proxying back to HAProxy. Your solution is really nifty though because there aren't two accepts going on.

In the thing that we're trying nginx terminates TCP (and SSL if needed), proxies back to a unix socket where HAProxy is listening. This gives us the best of both worlds: we can reload HAProxy all day long and socket listeners are hitless, and we rarely need to reload the nginx config which comes out of the box with hitless TCP sockets. But it's not quite as cool as this solution because HAProxy and nginx both have to accept.

We haven't decided that we're going to do our internal load balancing this way yet because the listening ports change so often, but if we do I'll probably be working on https://github.com/airbnb/synapse/pull/203 to do it. The PR makes it so that Synapse can manage both nginx and HAProxy simultaneously (to deal with the extra system complexity of an additional proxy). Or we'll try this cool thing out :-P

[bogomipz]

Can you remind me again why HA Proxy isn't hitless? Does it have to do with nproc?

[jolynch]

When HAProxy reloads, it has to re-bind the sockets that it was listening on. Due to a bug in how Linux implements listen port sharing (SO_REUSEPORT), new incoming connections can get dropped for a very brief (~1ms) period while HAProxy reloads. The Github article and the linked Yelp article both go into detail on this.

The tldr is that in Linux right now there is no way to gracefully drain connections from a listening socket. Many programs work around this by passing the socket file descriptor from the old process to the new process, which is, for example, how nginx works.

[bogomipz]

Right, the 3 way handshake can go sideways during that interim. Thanks.