[keeperofdakeys]

If you use an agent, just make sure you enable the option for it to prompt you upon use. This shouldn't require you to reenter your passphrase, so it's still unlocked in memory.

If you don't do this, any root user on any machine you connect to can use your ssh-agent connection to auth to other machines.

[jlgaddis]

I may be mistaken, but I think that applies only if you're using agent forwarding.

[keeperofdakeys]

Yes sorry, that's correct.