So standardize on a non-standard port, document it and when you do have ssh probes assume it's targeted rather than just another l33t child running du-jour crack(x).
Moving to non-standard means more sophisticated discovery
and possibly a more sophisticated attempt.
If your environment contains multiple ssh servers running on non-standard ports you would not notice when something out of the ordinary happens.
Something listening to an unexpected port should be something you must look into, not shrug at.