Wayland fixes that and is rapidly phasing out Xorg. It is long understood that X is not secure in this kind of scenario, it is NOT long understood that tmux isn’t (Or at least, I certainly have never heard this.)
That threat model essentially prohibits "tmux attach", which allows an attacker running as your user to connect to your terminal session, so I don't think it's a particularly useful threat model here. That's basically exactly what we signed up for by using tmux.
Running on a server doesn't change anything, you'd need to be running on a server where you routinely give people who shouldn't have root access, access to an account with sudo privileges with a password. And be relying on your attacker to not say, simply put aliases into your shell, replace your shell, modify your path, add an LD_PRELOAD, ptrace your processes, etc.