So you also need to make sure that your phone's browser doesn't have your Google password stored, and/or your phone's storage is encrypted with a strong-enough key.
Everytime I go to https://www.google.com/settings/security and click on 2-step verification, I'm required to enter my password if I haven't done so in the last 5 min or so.
With this scheme someone can't access your account by stealing your phone. You also can't access your account by getting your phone number to point to your new phone though.