I was talking about the GPG agent, so that the key on the smart card can be used to for sudo elevation on the remote host. This usually requires user interaction with the key, so just having access to the agent wouldn't do much. I don't think the ssh agent would help with this.
To your point, I wonder whether that consideration holds when the private key is held on an external device, like is the case with a YubiKey. I use that setup, and I can't add the key to the ssh agent.
$ ssh-add .ssh/id_yubikey_gpg.pub
Error loading key ".ssh/id_yubikey_gpg.pub": error in libcrypto
Don't these apps just use PAM? Since the initial complaint was about sudo, I'd figure pam / polkit would handle this, and apps would call those to obtain privilege elevation.