The source code is not the only condition for security. However it drastically decreases the threshold for the audits. People can even make a crowdfunding campaign and pay to professionals like it was done with TrueCrypt.
But even without such a campaign, evil developers would be in a constant danger that someone may discover a backdoor. It is a very unstable situation: just one person is enough to make a lot of noise, and everyone could be this person. And yes, people do read the sources: