http://www.bootc.net/archives/2013/06/09/my-perfect-gnupg-ss...
Coupled with a standard yubikey+gpg agent setup
Maybe look at my dotfiles if you are stuck:
- https://github.com/aviau/dotfiles