I like this solution, except for a couple of things. One, what happens if I lose my Yubikey (or it gets stolen)? Another, more realistic, problem is what do I do if I don't have my Yubikey (at my parents house)?
I have two Yubikeys, each with a separate SSH key that's been generated/signed by an offline master that resides on a USB key in a safe. In the event I lose one, I simply revoke/expire the key on keyservers and generate another using the offline key.
If you don't have your Yubikey at your parents house, maybe you should just enjoy spending time with them, instead of working on things that need an SSH key?
If you don't have your ATM-card, you can't withdraw money from your account. If you don't have your car keys, you can't use your car. This is completely desirable.
Yubikeys are very durable and small; just put them on your (physical) key-chain.
To prevent losing access when you lose your key, ensure you have a backup yubikey, or backup the keys stored on the yubikey so you can create a new one. If you use a solution where the secret cannot leave the key (such as Fido U2F keys) than register your backup key with all the services you use as well, or generate recovery codes for each service. Needless to say, these should be stored in a very secure place.