[wjossey]

Question for the OP-

I haven't ever worked on chat services, so this may not be reasonable. Would it be possible to use some other termination endpoint that sits in front of the service, that allows you to maintain persistent connections to the clients, but make for more transparent swaps of backend services?

So, for example could you leverage nginx or haproxy as the "termination" point for the chat connection, with those proxying back to the kubernetes service endpoint, which then proxy back to the real backend service. So, when you go to swap out the backend service, nginx / haproxy start forwarding to the new service transparently, while still maintaining the long-lived connection with the client.

If this was doable, it would mean you'd only have to drain if you needed to swapout the proxy layer, which is likely a less-frequent task, and thus allows you more agility with your backend services.

[bdimcheff]

Yes, as lotyrin pointed out below, the backend is already effectively an XMPP <-> Websockets bridge. Every time a user logs in, one way or the other we need to establish a connection between the websockets backend and the XMPP backend. The backend does do other things besides simply proxy, and we certainly could separate out the part that needs to keep the state. This deployment strategy is effectively a way to avoid having to do that work, at least for now.

[jkarneges]

This is essentially how Pushpin (http://pushpin.org) works. It can hold a raw WebSocket connection open with a client, but it speaks HTTP to the backend server, and the backend can be restarted without the client noticing.

[codegladiator]

This is a nginx module for this functionality

https://nchan.io/

I have used it before. Super easy to setup. Even with kubernetes.

[jkarneges]

Interesting, I didn't think Nchan would be able to drive a raw WebSocket session with the client, but upon closer look it seems like it might be possible using the auth and message forwarding hooks. Very cool.

[lotyrin]

It sounds like the product is already a proxy (Websocket -> XMPP) so I'm not sure what exactly they're deploying multiple times per day.

It also seems like they could have done simple blue/green and extend their websocket protocol and client to support a hand-off "hey, there's a new version of the proxy, reconnect in x seconds" message (and have idempotency of messages) such that they could have a rather smooth schedule where everyone can reconnect then disconnect gradually over some period of minutes instead of hours and not have either sudden spikes or any period of interruption for clients.

[ah-]

Or just a generic reconnect in case the websocket connection breaks for any reason really.

[lotyrin]

That would require an interruption in service between the server closing the old connection and the client reestablishing a new one, which they sought to avoid.

[vitalus]

In practice you probably would want pretty thin proxy layer as you said, which is then forwarding requests to other services as needed...but you would still need to re-deploy this proxy layer, and would thus need a similar solution to the one proposed in the article.