I agree, that's why signed source code releases are the safest thing you can get.
Keepass has signed releases (including the source code archive) that can be checked with OpenPGP.
If you trust the signed source code there's no reason you shouldn't trust the signed binary - unless you have sufficient time and expertise to audit the source.
* Being open source protects against a malicious developer. Otherwise there is nothing preventing him to build the binary with a different source, and send the passwords to his own server.
* Signed code archive prevents against a compromised hosting site.