In essence, without knowing much more about your setup, you're just shifting responsibility somewhere else.
Whitelisting is the route most (no actual statistics to backup, but based on personal intuition) would take because the server can prevent a connection after making a authentication (online-mode=true) and authorization (white-list=true) check.
Mesh like VPN solutions used to be popular in the past with software like Hamachi but, at least in my experience, performance was dismal and required additional setup for potentially non-technically minded end users.
Is white listing users the way to avoid it? I’m possibly too cautious and only have a couple of ports allowed though.