How is that? Everybody living in my house can get my Yubikey yet doesn't know my password. If I get robbed, my bank account is still (relatively) safe.
There are a lot more people far away from you than there are close to you. If breaking your security requires physical proximity (such as to steal a yubikey), then you are much safer just based on this. It's also easier for people to blindly steal credentials for millions of people online than it is for them to steal millions of physical security keys.
Alternatively, passwords are commonly reused across websites, so a failure of any of those websites can lead to a compromise of all of them, which is not the case with a YubiKey. Along that same line of thought, passwords are phishable, where YubiKeys are not.
It's also possible that people in your physical proximity could shoulder surf your password, install a keylogger (which could be a physical keylogger, if you normally use a USB keyboard, not just software), or use a strategically positioned camera to do some digital shoulder surfing. Passwords aren't immune to trust issues when it comes to physical proximity. Ideally, you trust those you are near to some extent.
YubiKey also has a fingerprint-protected device coming out soon[0]... which would raise the bar for the threat model in this discussion some. Using a fingerprint and/or PIN to unlock a YubiKey preserves most of the benefits, while eliminating most of the concerns that people are mentioning. HSMs can choose to self-erase after a certain number of failed PIN attempts, so even a short PIN is not something that can easily be brute forced without an unpatched vulnerability.
If websites would allow you to only use any one of your YubiKeys to authenticate (obviously meaning you can have multiple, with backup YubiKeys stored somewhere safe in case you lose your main one), I think that would be a significant improvement in security over password authentication for most people. This is basically what the WebAuthn standard is attempting to do. I don't expect most people to be interested in buying 3 security keys and carrying one around all the time, though.
For the last bit: If it's suitably seamless, it's actually not that bad. I've been carrying one on my keyring, and it's just another key, only this one "unlocks" websites.
Most people in your home are not trying to hack you.
A lot of people outside your home are trying to hack you.
Shifting your exposure from "everyone in the world with an internet connection" to "people who are in/near your home" greatly reduces your risk, objectively.
There are a lot more people far away from you than there are close to you. If breaking your security requires physical proximity (such as to steal a yubikey), then you are much safer just based on this. It's also easier for people to blindly steal credentials for millions of people online than it is for them to steal millions of physical security keys.
Alternatively, passwords are commonly reused across websites, so a failure of any of those websites can lead to a compromise of all of them, which is not the case with a YubiKey. Along that same line of thought, passwords are phishable, where YubiKeys are not.
It's also possible that people in your physical proximity could shoulder surf your password, install a keylogger (which could be a physical keylogger, if you normally use a USB keyboard, not just software), or use a strategically positioned camera to do some digital shoulder surfing. Passwords aren't immune to trust issues when it comes to physical proximity. Ideally, you trust those you are near to some extent.
YubiKey also has a fingerprint-protected device coming out soon[0]... which would raise the bar for the threat model in this discussion some. Using a fingerprint and/or PIN to unlock a YubiKey preserves most of the benefits, while eliminating most of the concerns that people are mentioning. HSMs can choose to self-erase after a certain number of failed PIN attempts, so even a short PIN is not something that can easily be brute forced without an unpatched vulnerability.
If websites would allow you to only use any one of your YubiKeys to authenticate (obviously meaning you can have multiple, with backup YubiKeys stored somewhere safe in case you lose your main one), I think that would be a significant improvement in security over password authentication for most people. This is basically what the WebAuthn standard is attempting to do. I don't expect most people to be interested in buying 3 security keys and carrying one around all the time, though.
[0]: https://www.yubico.com/blog/yubico-reveals-first-biometric-y...