[avian]

In my experience a SSH service on a random, non-standard port gets surprisingly few probes. I look after several machines and I see less than one attempt per year (versus hundreds per day for port 22). I have yet to see somebody probe a SSH that listens on IPv6-only.

[pmoriarty]

You might want to consider using port knocking[1] to make your ssh server even less susceptible to attack.

[1] - https://en.wikipedia.org/wiki/Port_knocking

[semi-extrinsic]

No, the best solution is to only allow login by SSH keys. No passwords => brute-forcing is impossible. So your threat model for someone gaining access no longer includes someone using weak passwords.

[pmoriarty]

If your ssh port is wide open and there's a remotely exploitable vulnerability, then using keys may not save you.

But there's no reason you couldn't use both keys and port knocking at the same time.