[leejoramo]

If you don't use passwords on your keys, then any one who can access the key files on your local system, could access the remote server. Not using passwords removes an important layer of security.

That said, often you need to have password-less keys in order for systems to automatically communicate. For example, I setup backup servers without passwords.