I do understand your concern. And to answer your question, I don't care if they do access my physical letters since they can do that without my permission anyway. It's just a matter of their will and tech they will use.
We're talking about a different "them" though. I believe you're talking about the government, their capabilities and permissions are quite clear. Here, we're talking about a third party though who you conciously opt in to.