[count]

I use the gpg-agent option as well, but hate it (GPG makes me feel icky). That is not, as I understand it, using an X.509 key for authentication though. Which is what I'd like (ssh-agent forward an X.509 based auth mechanism - like remote PKCS#11 over that agent connection, so the key stays on my yubi, and I only need CA certs on the remote hosts.

[rkeene2]

This is exactly what I do -- I posted some links to both the SSH Agent (well, the ChromeOS version -- I've got C versions as well) and the PKCS#11 module which talks to the SSH Agent. So on a remote system I can do things like sign files using PKCS#11 and passwordless sudo using pam_pkcs11.