For a little while, I used a YubiKey NEO-n as an OpenPGP smartcard, with gpg-agent running as an ssh-agent (so that my SSH key was only present on my smartcard).
While great in theory, the authentication time made it totally impractical to connect to multiple servers at once.
I know this doesn't invalidate the general idea of using gpg-agent as ssh-agent -- just an anecdote.
Thinking on this again now, perhaps using a shorter key (I'm sure I would have chosen 4k RSA) would have helped. Regardless, I did find that connecting to 50+ hosts at the same time, most of them would hit my 3 second connect timeout with the neo-n and gpg-agent.
This is a somewhat unusual use-case, I will grant, but I also found a somewhat-noticeable delay in connecting to a single server, on the order of several hundred milliseconds.
I will not give up on this setup, minus the Yubikey (would love to one day), but it recently broke it when I updated via F-Droid, and it has not been the same since.
author here: All of that should be on Part 2, in a week or so.
Part 2 will cover emails (Thunderbird, mutt), pass as a password store, setting up OpenKeychain + a Yubikey Neo on an Android phone, K-9 Mail on Android, Yubico authenticator for Android as a 2-step auth, ssh, and Keybase. Future parts would cover hdd encryption via LUKS, authentication for sudo and more, GPG by NFC on your desktop, GPG intents to open doors, etc.
While great in theory, the authentication time made it totally impractical to connect to multiple servers at once.
I know this doesn't invalidate the general idea of using gpg-agent as ssh-agent -- just an anecdote.