That is not the only time you use passwords over ssh, e.g. I don't use a password to remote into my desktop from my laptop, but I do use one when using sudo on the desktop.
Actually this is something that is relevant to my interests.
I prefer to have sudo ask for a password when I'm physically in front of the machine, but not if it's a remote session (e.g. SSH from my laptop to my desktop).
Maybe the SSH agent on the client can re-authenticate to the server when requested?
Note that this is a bad idea from the security standpoint, as it requires SSH agent forwarding. Which means that, if the remote server is compromised, the attacker can use your SSH agent to log into other servers as you.
I was talking about the GPG agent, so that the key on the smart card can be used to for sudo elevation on the remote host. This usually requires user interaction with the key, so just having access to the agent wouldn't do much. I don't think the ssh agent would help with this.
To your point, I wonder whether that consideration holds when the private key is held on an external device, like is the case with a YubiKey. I use that setup, and I can't add the key to the ssh agent.
$ ssh-add .ssh/id_yubikey_gpg.pub
Error loading key ".ssh/id_yubikey_gpg.pub": error in libcrypto
Don't these apps just use PAM? Since the initial complaint was about sudo, I'd figure pam / polkit would handle this, and apps would call those to obtain privilege elevation.
FWIW, you can probably configure sudo to use something other than passwords. On a Mac you can use the fingerprint reader for example, it's just disabled by default.
And your terminal may come with a password manager too, which would be unlocked with whatever means.
Again, on a Mac with iTerm you can do this with a fingerprint.
I prefer to have sudo ask for a password when I'm physically in front of the machine, but not if it's a remote session (e.g. SSH from my laptop to my desktop).
Maybe the SSH agent on the client can re-authenticate to the server when requested?