I've configured ssh to use a different key for each server [0]. I don't bother with passphrases on each key, instead relying on a encrypted home folder. Also, I have a script [1] for rotating keys, so rotating all my credentials is a single command.
What good will an encrypted home folder do if your computer gets compromised? It protects against offline, physical attacks, sure, but I fail to see how it is safe against the myriad of remote attacks there exists.
It doesn't. But neither do passphrases. The key has to be in process memory at some point, and if you're running a compromised process under your user, you've already lost.
The only threat a passphrase protects against that an encrypted home doesn't is someone walking up to my unlocked laptop (it locks automatically after 5 minutes, and I make a point of locking it when I'm leaving). If you run ssh-agent, you're exposed to this threat anyways.
EDIT: After some thought, passphrases can protect against a poorly executed attack, where just files are copied wholesale, or an accidental leak. So while theoretically you're not protected, it may be sufficient for some practical security. I'll be looking into using ssh-agent in the near future.
A passphrase also protects against someone acquiring your private key by some other means. For example, if you accidentally copy the key file somewhere, or some tool (Dropbox, OwnCloud, any cloud syncing tool) syncs it to an unsecured or compromised location. The passphrase adds a knowledge factor to the key's security.
Passphrases on keys really only provide some incidental security there, for keys that you don't happen to unlock while compromised. (Or the adversary might neglect to intercept your passphrases.)
Use a yubikey or smart card to level up from these.
Something like Qubes OS is better defense against remote attacks, there is a lot of other mayhem that kind of compromise might cause. (Also passphrases don't really help there).