You might get pwned by (1) the government, (2) your mobile carrier, or (3) a hacker that can social-engineer your mobile carrier's tech-support (SIM jacking / SIM swap attack).
Yes, that's true, but it's not an argument in favor of authentication based on something I have.
I don't think we can prevent everything, but I at least want there to be some way to undo the damage (things like courts, chargebacks and so on).
I think the argument for something you have is cyber-physical security. No matter how advanced malware is it won't be able to extend a finger through your monitor and tap the capacitive touch sensor of your Yubikey.