[upofadown]

Put a PIN on it...

You can remove the PIN later. The hardware key can be set up to wipe the device after, say, 10 incorrect guesses.

[taeric]

If it can be wiped on incorrect pin usage, you can be maliciously locked out of accounts relatively easily.

[upofadown]

Then you are going to know that the friend was messing with your hardware key. Better that than finding out they are willing to betray you by stealing from you.