Quite the contrary, actually: not using a browser extension makes you much more susceptible to phishing attacks, since your password manager won't be able to protect you from copy-pasting credentials into an imposter website.
You don't need to compromise the extension but that sure is another drawback of installing more software than actually needed. You could exploit the password manager extension from inside the browser and that way get access to the password manager since you created a direct path to it weakening the otherwise strong browser security.
The browser should stay isolated and seperate from anything on the device instead of integrating "dog doors" in the software with the no1 biggest attack surface of any modern device.