I and some discord friends use a matrix server as a backup + place for convos involving PII but we only really do text and images. Depending on your needs it might fit for you too.
I set up a matrix server with voice chat for myself and a friend but hosting the TURN server through NAT is tricky and (perhaps it was my stack) it didn't always connect (one of us would click the call button and the other person may or may not get a call notification, and clicking answer may or may not actually connect the call). I'm no expert so maybe there's something I did wrong.
TURN + NAT is always going to be a pain. If IPv6 isn't available to you or all of your friends, going for a VPS seems like the best choice to me. Just don't go for the extremely expensive server providers (i.e. Amazon) if you care about affordability.
This will cost a few dollars per month but it makes hosting UDP services becomes a lot easier.
Text chat + images work quite well though.