If you choose only to authenticate with your phone then you can't authenticate without it, yes.
If you "always" have your phone this probably doesn't feel like a big deal unless somehow Yahoo Japan is also their international embassy and gateway to access basic services, unlike the Yahoo! I'm familiar with.
If you "always" have your phone this probably doesn't feel like a big deal unless somehow Yahoo Japan is also their international embassy and gateway to access basic services, unlike the Yahoo! I'm familiar with.