[mb0]

Something as simple as an exploit against a php script could lead to unprivileged code execution & something as minimal as memory exhaustion could kill off your sshd process. If the sshd process is killed, and the attacker knows what port it was on (netstat -na | grep LISTEN), the attacker could spawn their own daemon on the unprivileged port that was previously being used for sshd, and that daemon could be used to collect plain-text passwords that people try to use when logging in. Heck, they could craft their daemon to log all keystrokes & which could grab the local root password.

[telephonetemp]

This is a very valid point and a good reason to choose a port in the privileged range if you use an alternative port for SSH. I wonder, though, what is the worst it can do if you don't use password authentication? E.g., to offer you a fake shell and try to capture your sudo password a rogue daemon would have to have access to your public key, which as an unprivileged user it shouldn't.