[dredmorbius]

That key by itself becomes a valid access token.

If someone snarfs your key from local storage, they are you. They don't need a 2nd secret (the passphrase) to unlock the key.

If you use methods such as ssh-agent, you get all the convenience of a passwordless key (save the entering the passphrase into the agent) without the security risks.

You may find it necessary to use passwordless keys for some server processes, say, Nagios authentication or running remote jobs between servers. So long as you isolate these keys, restrict access to known hosts / IP ranges, etc., you're fairly well covered. Forced commands are another option to reduce the risk of such keys, though these aren't always appropriate.