[gnufx]

I can't tell what that involves but, for instance, the two GUI things I typically use with SSH are Emacs (openssh) and x2go (libssh), and they don't do that. Surely you want the agent anyway.