One solution is reproducible builds and signatures.
The bitcoin community, for example, uses Gitian to reproducibly build bitcoind. Both Bitcoin Core and Bitcoin Classic host repositories with signed hashes for the output of those builds:
(As I understand it, several Altcoins do the same as well)
Anybody can follow the published guides for how to perform such a build, and compare their results with the published ones. Because the published hashes are signed, you have a reasonable degree of certainty that a variety of people are involved in the process, which also gives you a greater degree of confidence in the quality of the binary releases even if you don't want to compile it yourself (and, if you do compile it, you're free to add to the consensus that the binary build is good by PRing your own results)
Reproducible builds do not cut it. If you have a known good binary signature, you'd also have a known good source signature and wouldn't have the problem.
Or, to put it in better words, where do you get the certificate to check your build from? At extreme paranoia levels, you simply can never be sure you have the same software as everybody else, thus the only safe alternative is reviewing your copy yourself.
(How do you know the computer is showing you the correct contents of your files? Didn't think that well enough yet.)
Diverse Double Compiling is a proven solution to the On Trusting Trust problem [0]. So, if a package maintainer signs a package and posts that signature on an https page, I can have a high level of confidence that the software I compile and run on my machine is identical.
Here is some advice from Schneier on running secure software against a state-level adversary [1][2]. However, even that is not immune from a black bag job [3].
The bitcoin community, for example, uses Gitian to reproducibly build bitcoind. Both Bitcoin Core and Bitcoin Classic host repositories with signed hashes for the output of those builds:
https://github.com/bitcoin/gitian.sigs https://github.com/bitcoinclassic/gitian.sigs
(As I understand it, several Altcoins do the same as well)
Anybody can follow the published guides for how to perform such a build, and compare their results with the published ones. Because the published hashes are signed, you have a reasonable degree of certainty that a variety of people are involved in the process, which also gives you a greater degree of confidence in the quality of the binary releases even if you don't want to compile it yourself (and, if you do compile it, you're free to add to the consensus that the binary build is good by PRing your own results)