[jordemort]

No, the private key never leaves the YubiKey. SSH and GPG talk to the YubiKey as if it were a smartcard - data is send to the key to be signed or encrypted.