Not if I'm on your device and hijacking your already-authenticated connection. I just need to be careful enough to do it in the background in such a way that you don't notice.
If my device got stolen I would remove the device from my accounts immediately. And without the second factor you wouldn't be able to do anything about it.