There is technically one minor one, which really isn't one - but you should be aware of.
Someone can take your authorized keys and add them to a box they control, and trick you into logging in.
However, this would trigger the "new host" warning SSH gives you, and you can minimize this by minimizing which hosts you allow your private keys to be used on.
And if someone is so actively trying to attack you they probably have more direct methods available.
What? How? What does putting my authorized keys file on another host do in terms of tricking me to log in? Authorized keys only matters on the host you are using when you type `ssh <some.host>`. The ssh client compares the public key of the remote host to the list in your `authorized_keys` file and, only if there is a match, skips serving you TOFU.
EDIT: I mixed up authorized_keys and known_hosts. But, the remote server doesn't need your authorized_keys file to grant you access so not sure the visibility of authorized_keys matters.
Someone can take your authorized keys and add them to a box they control, and trick you into logging in.
However, this would trigger the "new host" warning SSH gives you, and you can minimize this by minimizing which hosts you allow your private keys to be used on.
And if someone is so actively trying to attack you they probably have more direct methods available.