[jvoorhis]

Search for "ssh agent forwarding attack" or similar.

[yoo1I]

I am glad you specified "or similar"

http://imgur.com/768xaL3

But you're correct, an attacker on the remote system with access to the socket (i.e. usually root) can steal your identity.