i think the main argument for leaving it default is [1] for any additional/new sysadmin that has to touch the system. but imo this goes both ways; i'd rather not have my house key hidden under my doormat, where every burglar can check for it in a few seconds. if i taped the key to a random tree branch (even in plain sight), it would be a much safer bet.
If you're going to insist on a security analogy involving keys and doors in the real world, a better one would be that the attacker has to search every single atom of the doormat to find the key. If you have PubKeyAuthentication enabled and PasswordAuthentication disabled, the world will end before the attacker finds your key (in expectation).
Changing SSH to a non-standard port is an inelegant solution and doesn't actually bolster security.
So standardize on a non-standard port, document it and when you do have ssh probes assume it's targeted rather than just another l33t child running du-jour crack(x).
Moving to non-standard means more sophisticated discovery
and possibly a more sophisticated attempt.
One reason is because setting it to a random high port allows non-root users to turn on the SSH server if the admin turned it off. Probably not super likely to happen, but it's possible.
[1] https://en.wikipedia.org/wiki/Principle_of_least_astonishmen...