My approach aswell. Lock down ssh-agent and restrict its usage as much as possible. Securing your keys is also very reasonable but it cant silence this naging voice in the back of my head that keeps reminding me of a compromised ssh-agent or shell, whenever i authorize privileged actions.