Your address isn't a secret, and the "something you know" part of TFA isn't supposed to be common knowledge. To stretch the metaphor, your address is your user name, not the password
It could be the same with a phone (username / phone). I figure between my phone having a password and (I assume) the fact that I would be able to revoke authenticator access to a device quickly I would be comfortable with the risk. I honestly hope this catches on because I would prefer it to the current system of having to manage a bunch of separate passwords (which currently are also saved on my phone)
this is only fine if you can trust your own phone and that there is no malicious apps doing something, I don't really trust only my password, same as I don't only trust my phone, thus I prefer 2FA for really secure things as tied combination, both factors have weakness