[denrober]

My only thought is even with password auth. turned off hostile persons can still connect to the ssh daemon and might discover and unknown vulnerability or some type of dos. If you know someone is trouble why let them in even if you are wearing armor?

[numbsafari]

It'd really suck if they could forge traffic from you that looked like a failed login attempt and that resulted in you getting banned.

[anderiv]

That is why fail2ban allows you to configure a whitelisted IP (your office, VPN, etc.) that will never get banned.