[sureglymop]

So if I have access to the agent and can authenticate with it at another remote location, then if I control that remote location I should be able to grab the key at _some point_ during this process. I may have to implement my own ssh server to do so but it should theoretically be possible, no?

[semi]

Not as far as I'm aware, at least not without some really clever attack. The design is such that all you're doing is asking the remote agent to use it's private key to sign things, it never actually exposes the private key. Your agent doesn't even always have access to your private key- it could be on a TPM or Yubikey where the key never leaves the device. you are just passing around a connection to your agent that is able to authenticate with a private key.

I would not be surprised if there is some way to attack this into getting a private key, but it would either be a direct attack on the agent code like sending it malformed messages to somehow get remote code execution to then read the key. Or some more complicated attack on the cryptography where you repeatedly force it to auth and can somehow use the results to reduce the key space needed to brute force the key - along the lines of a known plaintext attack.

But I'm also just a hobbiest here who has looked a little into the security model but I am by no means a cryptography expert, so take all of this with a grain of salt.