[pdpi]

One solution is reproducible builds and signatures.

The bitcoin community, for example, uses Gitian to reproducibly build bitcoind. Both Bitcoin Core and Bitcoin Classic host repositories with signed hashes for the output of those builds:

https://github.com/bitcoin/gitian.sigs https://github.com/bitcoinclassic/gitian.sigs

(As I understand it, several Altcoins do the same as well)

Anybody can follow the published guides for how to perform such a build, and compare their results with the published ones. Because the published hashes are signed, you have a reasonable degree of certainty that a variety of people are involved in the process, which also gives you a greater degree of confidence in the quality of the binary releases even if you don't want to compile it yourself (and, if you do compile it, you're free to add to the consensus that the binary build is good by PRing your own results)