As much as I like the idea of Gmail/mail "extensions" and services, as long as I can not run them locally and fully control my data, they're a no go for me.
After all, once I've granted them the permission to use the Gmail API for my account, they can do pretty much anything with my email. Would you give any third part access to all your physical letters without being able to check what they do with them?
I do understand your concern. And to answer your question, I don't care if they do access my physical letters since they can do that without my permission anyway. It's just a matter of their will and tech they will use.
We're talking about a different "them" though. I believe you're talking about the government, their capabilities and permissions are quite clear. Here, we're talking about a third party though who you conciously opt in to.