[tombh]

I think we all understand the usefulness of a road-warrior-style VPN. But it doesn't seem so clear what k8s is adding here?

Anyway, on the topic of scalable UDP services, does anyone have any experience of load balancing a UDP service? Because UDP is connectionless there's no obvious way to make UDP packets "sticky". Are there any established practices that could help scale this k8s Wireguard service to 2 or more containers?

[sclevine]

I'm just using K8s (specifically: K3s) for configuration management in this case. This post hits the nail on the head: https://news.ycombinator.com/item?id=23006114

That said, NGINX can do UDP load balancing and WireGuard is stateless, so it should be possible to use this with a Service + NGINX ingress controller at scale: https://kubernetes.github.io/ingress-nginx/user-guide/exposi...

I have not tried it though.

[georgyo]

Load balancing UDP isn't too difficult. However that is not the hard part here. It is ensuring the routing happens correctly.

A client must hard code it's IP address currently, which means if it can connect to more than one node, then it is unclear which path a response from a server should take to get back to that client. Each VPN instance could run NAT, but then users would never be able to talk to each other.

Wireguard makes this significantly harder than say ipsec. WG has nothing to indicate when a client connects. And there is no dead peer detection, so you cannot tell one a client disconnects. IE. Scripting something to update a global routing table to say which sever has which client is near impossible.

I use wireguard daily for personal stuff. However I cannot think how I would make it work in an active-active situation besides NAT, which I don't want.

[yardstick]

WireGuard proponents would probably tell you to run BGP or some routing protocol over the VPN, maybe GRE too?

I agree with you, WireGuard makes this significantly harder than it needs to be. Other protocols do better in this respect.

[KenanSulayman]

Well if you make it a DaemonSet you could technically use the container as the network interface of other containers throughout the whole cluster. That said, I'm very happy that his example k8s deployment uses secrets.

I didn't know Ubuntu 20.04 back ported WG into its 5.4 kernel. I spent a few hours yesterday fixing a node after breaking ZFS because I upgraded to 5.6 for WG support. I feel rather silly now..

edit: rektide mentioned 'kilo' which actually does exactly what I said (https://github.com/squat/kilo).

[tombh]

That's an interesting idea about using a unified network interface. Do you know how you might then get the right packets to the right containers/processes? Does that even matter with Wireguard?

[KenanSulayman]

You can use a different container as network in docker: 'container:<name>' would route the container traffic through specified container.

Example vpn container:

  docker run --name foo --cap-add=NET_ADMIN ...

Other container:

  docker run --net=container:foo ...

Now you'd need to specify the respective routing rules [1] in the container.

[1] i.e. https://github.com/bubuntux/nordvpn/blob/master/start_vpn.sh...

[dmw_ng]

This has been done for 2 decades or more by hashing the connection tuple somehow, e.g. hash(src ip | src port) % number-of-replicas, etc.

Naturally it breaks if replica count changes.

The other option is conntrack but then you have another stateful component that doesn't scale

[tombh]

That makes sense, though I suppose for a road warrior setup the source IP might change every so often right?

[ignoramous]

Wireguard, inspired by MoSH, handles reconnections especially well. I guess, TCP flows tunnled through UDP might be reset depending on which server (behind the load-balancer) is handling them?

Cloudflare shared, in some detail, how they load-balance wireguard traffic for roaming-ip and ports: https://news.ycombinator.com/item?id=21070315

Usually, I've seen UDP client-affinity set on (source-ip, destination-ip) tuple to handle port changes, but it doesn't help client with roaming-ips.

[dfox]

As for what k8s adds here I don't know, but this thing adds to k8s knowledge one interesting fact: It can be useful to run container that does not contain any process doing useful work ;)