I'd say it is two factor (googles implementation, the attack is classical social engineering): something you know (the password) and something you have (access to your phone).
In Google's implementation, only the "something you have" is really necessary for access. If you have the phone but not the password, you can just issue a password reset, which is confirmed via the phone, so the password doesn't function as a second factor independent of the phone.
I wonder if anyone implements the restriction that a password reset can only be ordered after a certain time (a week, say) since the last successful password entry, for long-established accounts. Most real password resets are likely either in long-dormant or recently-created accounts, and this would add just another layer of partial protection against these kinds of attacks.
Whoa. I hadn't realized this. So someone that knows my email address and has my phone has access to my entire life, because all password resets use my email address.
If it's come to this, to using "something you have", then we can all go back to using paper password notebooks. They offer the same security, surprisingly.
There have been many attacks in Russia where attackers would get a duplicate sim card from the operator (either with a bribe or a forged passport) and then proceed to hack everything linked to it - gmail, banks, facebook, etc
At that point (identify fraud) there's very little you can do short of not allowing password resets. Even if you have a human customer support rep to verify your identity prior to resetting a password, you can just send your forged identity documents.
It think this is something that Google is working on. By capturing multiple "facts" about you a company can be fairly certain who is trying to access an account, apart from passwords and phones. For example, they can tell if you spend 99.9% of your life in one country and then suddenly request a password reset from another country. I'm sure there are lots of other minor indicators that are much more difficult to fake, but putting them all together they should be able to get a pretty good picture of who you truly are even without proper 2FA.
Probably the only thing you can do is to have a separate secret email or phone only for account recovery, which are not linked to you in any way, or recovery codes printed out and stored in a secure location