Changing the ssh port to something in the 50_000 range drastically reduced the number of attempts and left my logs much cleaner :).