EDIT: a better solution would be if the store itself allowed you to inspect the source that went into building the plugin. Then you would only need to trust the store itself, which you already do (when you trust the browser).
It's quite common among many groups of people to download and install locally as it also protects you from unwanted automatic updates. For instance, those using MetaMask or Scatter to interact with a blockchain are often advised to install the extension offline.
EDIT: a better solution would be if the store itself allowed you to inspect the source that went into building the plugin. Then you would only need to trust the store itself, which you already do (when you trust the browser).