Keep in mind the danger that if the SSH server crashes other non-privileged users on that box can launch a fake server on that >1024 port to take its place.
But unless the non-privileged users have access to the ssh key files - definitely not allowed in any sane set-up - their MITM sshd will be throwing big, obvious error messages at most of the users. (Which is the mechanism protecting you against MITM's via all sorts of "intercept the packets" network attacks.)
My servers only have `root` and my own sudo user, (and other default system users). I also run all apps on Docker. I don't think this would be an issue for me.