Sooo you need to trust someone that does have that experience to do the verification. What alternative are you suggesting? Is there some cool way to write your crypto so that a layman can successfully verify the integrity of a binary?
One solution might be if some big corporation or even a government, or why not Bill Gates himself, offered a big ongoing bug-bounty for this Open Source Software.