[darklajid]

No, because

- the password you store on the phone (in cleartext) is not the real account password, it's a string for this device only and you can revoke access at any time

- Someone should not be able to use your phone (via call/text message) as unlock device, unless you lost it

- In that case you should lock the SIM for for a multitude of reasons anyway - and you'll get a new SIM that you can use to recover your account

I think the first one is the most important though: You just don't have to store your real password.