You have to trust everyone who claims the binary is clean, yes. But the vendor must also trust all these conspirators! And they can't bribe everyone - it only takes one honest guy with a decompiler and it's game over.
This is not a case of "many eyes make all bugs shallow, so all OSS software is safe". This is a case of "many eyes will look at my covert code, can't take the risk of discovery". Debian SSL is not comparable because it was an accidental (if stupid) bug. We are talking about deliberate malice, not bugs (which are inevitable in open and closed code alike).
At the state actor level do you really think you will be able to distinguish a stupid bug and an intentional flaw? Its not like they will make it call home with a bunch of code, they would just use a class of attack not well known and you wouldn't be able to tell the difference. That's all it takes, one little hole.
I imagine certain organisations knew about buffer overflow bugs long before they were used publicly, so imagine if this was the 70's and you saw some strcpy calls peppered into some useful code, would you really be able to know 1) the class of attack exists and 2) if it was intentional or not?
Yeah, but that is always a threat with any code ever written by anybody other than oneself. The only assurance against that is if one writes their own code compiled with their own compiler and run on their own fabricated hardware. Oh, and implementing their own security algorithms. Which means any data exchange would be impossible.