[Nullence]

Welp, I have no idea if anyone will read this as its been a little while since it was put up and is in a flagged comment, but hey I'll do a lil explanation of some of the issues I ran in to, and how it works.

How it works:

In short, Wag adds an eBPF program to a WireGuard device that it instantiates. The eBPF program uses a number of hash maps and LPM (longest prefix matching trie) maps to determine the policies that are applied traffic coming in on the wireguard device. These policies based on the ACLs defined per user/group, and contain MFA/Allow/Deny rules which require mfa, allow without auth and deny always respectively.

Wag also watches all the wireguard peers ingress IP addresses, and when an address changes it deauthenticates the user and requires the user to complete a login challenge. This is done by basically setting a bit in the maps exposed to eBPF that says "unauthorised"

Challenges:

First and foremost with WireGuard there is no good way of determining if an "external ip" i.e where the user is connecting from has changed. There was a patch set submitted for review in 2022~ that was never actually added to the kernel that would have added netlink compatibility and thus event based notification that things had changed, but alas that was never reviewed by Jason Donenfeld and has quietly died the death.

Secondly was defining multiple policies per route was quite difficult as eBPF doesnt do dynamic memory even in userland exposed maps and I wanted multiple rules per route, i.e you might allow port 80/tcp when MFA has passed but otherwise always allow 22/tcp. So to do that I had to define a maximum number of rules that could be inserted as one memory blob into the LPM map that the ebpf program would then linearly search to make its decision.

Thirdly has been making everything highly available which has been a bit of an on-going battle with ETCd mainly around how it manages cluster certificates as they dont (as of 2024 but it may be coming soon) expose the right structures to allow for dyanmic certificate creation, so you have to kind of make a wrapper around that in order to get everything going.

Im sure there are other things that I've had struggles with, but these are what come to mind immediately!

[mbreese]

Thank you for posting this. These are the types of details that you just don’t get unless the author has really worked through these issues in depth.

Best of luck with the project!

[Nullence]

Thanks! It's been a bit of a labor of love for quite a while, these are the big three but there are a bunch of other little things.

Like the time I had to optimise map insertion because the linux kernel does some truly insane locking when you use specific types of eBPF maps:

https://github.com/NHAS/wag/issues/84

This is slated to be improved (or has already been improved in kernel 6.8?). But for now wag sort of just side steps it in a horribly stateful way.

[aragilar]

Ooh, this looks cool! I'm mildly curious given Windows apparently is adding support for eBPF whether you could also get this running on Windows as well.

On the patch, maybe try reposting it on the list, with the pointer to your project to see if that provokes a new review?

[Nullence]

Haha, me and a friend were both looking at the windows eBPF project and wondering if it might work. Im not really looking to support windows, but if someone comes to me and tells me it works I'll pop it on the read me