If someone "borrows" your thumb drive they could extract the secrets and return it without you knowing, but AFAIK secrets cannot be extracted from a real token such as a Yubikey.
Come on mate, just use a bit of imagination.
You use the pam_usb module to login locally and an encrypted passkey stored on the same usb thumbdrive for remote SSH connections.
From the webpage:
* Non-intrusive. pam_usb doesn’t require any modifications of the USB storage device to work (no additional partitions required).
* USB Serial number, model and vendor verification.