It's most likely easier to brute force a password than to break into someone's house. Would be easier to demand all credentials by gunpoint with that much effort.
That's a fair point, but that's not the only attack vector. I carry my token around on my keys which makes it vulnerable to being pick pocketed or just left behind somewhere. I think the original point was that you're just shifting your single authentication factor, not necessarily making it more secure. My key is only used for 2FA so even if someone were to get access to it, they'd have to know my password as well to get use out of it.