[Xylakant]

Absolutely, but there are various attack vectors that different mitigations are effective against.

The program doesn’t even need to be malicious, for a while it was a pretty common attack vector to trick browsers into uploading random file you could access.

Later, a malicious ssh server could read memory of the ssh process, potentially exposing the private key (CVE-2016-0777)

Using an agent with an encrypted key protects against that. Using a yubikey/smartcard as well. So it’s strictly a good thing to use it.

A yubikey could potentially protect you against a malicious program that wants to open connections if you have set it up to confirm every key operation - but that comes at a cost. You could also use little snitch to see what network connections a program opens, protecting you against a program trying to use your agent to access a server.