[telephonetemp]

This is a very valid point and a good reason to choose a port in the privileged range if you use an alternative port for SSH. I wonder, though, what is the worst it can do if you don't use password authentication? E.g., to offer you a fake shell and try to capture your sudo password a rogue daemon would have to have access to your public key, which as an unprivileged user it shouldn't.