WEP, WPA, WPA2... why keep reinventing the same wheel? Each new iteration inevitable turns out to be less-than-perfect and keep adding more and more complexity and overhead - for one, join/leave times keep increasing, up to a point where we have a separate standard (802.11r) just to get back pre-WPA roaming speeds (at cost of even more protocol complexity overhead).
Here's crazy idea: Why not run open network + IPSEC, or heck, even OpenVPN? Obviously, drop all non-VPN traffic right on the AP (or first router after it) to nip freeloaders in the bud.
IPSec and OpenVPN are complex like hell and not very well supported across OSes (think of the UI too). This is why I'm waiting for Wireguard.io, it looks like a step in a good direction.
I've been experimenting with this exact kind of network setup -- open WiFi that only allows packets to and from the WireGuard endpoint. The nice part is that it means I just have WireGuard on all the time, and because it roams, when I connect to a different WiFi networks, I remain on my home network automatically. I've also been putting my actual wlan0 interface in a separate namespace, isolated from real things. This way there's no chance of leaking data: https://www.wireguard.com/netns/#the-new-namespace-solution
Most captive portal routers don't block DNS (because they use iptables rules to handle authentication). That's why you can use iodine to proxy TCP-over-DNS on such APs. So if you just had an open access point, unless you provided no DNS servers except over VPN, people would still be able to use your AP.
Presumably you want to reply to DNS requests for a hostname for your captive portal. You might try to just use a raw ip address, but then you can't use https. So then you have the problem that you can't just reply with a fake answer for other domains due to caching. E.g. Windows caches negative responses for 5 minutes, which would be a pretty bad experience for your customers.
I guess you might be able to just fail to reply to DNS requests for domains outside you captive portal, I have no idea if anyone has tried that or there might be other complications.
Edit: Actually not replying wouldn't work great either because then the user can't be redirected to the captive portal. This might be less of an issue today since most devices have standardized a way to detect captive portals using a small set of hostnames.
Aside from custom root certs being installed, https is out either way, not only with IP address, right? HSTS makes that even more of a problem, and pinning makes even the custom root cert a problem. Captive portals seem like an increasingly fragile idea, except that OSes increasingly intervene appropriately.
...and bring up UI (sandboxed browser) if anything else than expected short 2xx was received.
Step 2) Mobile device reuse same cookies for the same portal second time around, so portal can recognize returning user and let them in without annoying with login prompt each time.
Assuming mobile device has a separate cookie jar for this, and captive portal is HTTPS with proper certificate, and doesn't try to break other people's SSL, that is pretty secure. Sure, this is not the most efficient protocol, but up-side is - it's open-ended. You can fit here anything from "press here if you agree to behave" to complete walled garden like hotspot on a train that shows you interactive map, schedule and transport connections available on the next stop along with small banner asking $1 for full internet access.
You just reply to any DNS request with your own server's IP, which accepts any HTTP requests with a redirection to the captive portal. The replies can have low TTLs to avoid the caching problem.
Newer captive portals (FON with latest firmware for example) are blocking iodine by being clever about which request to answer (bunch of A and AAAA with some of responses being CNAMEs, which are quickly followed by resolution of just received answers to A/AAAA), and which to block (bunch of NULL queries or "dangling" CNAMEs that are not followed by A/AAAA).
That being said - that's a bit over the top. The only reason I wouldn't want my AP to be open and unfiltered is - I don't want any junk being sent out which is attributable to me (by IP), and which I have no control over. Iodine, being a tunnel will only transport to "guest"'s server and go into the wider Internet with their server's IP.
If somebody is in enough of a squezze to jump over that sort of hoops, and there're no costs/risks for me - let them have it. I'm fine with that.
Here's crazy idea: Why not run open network + IPSEC, or heck, even OpenVPN? Obviously, drop all non-VPN traffic right on the AP (or first router after it) to nip freeloaders in the bud.