We've known that in theory your compiler et al can be backdoored but in practice we can feel a lot safer compiling our own software than using proprietary binaries.
I think that might be where we disagree. Without a competent security audit I think you are falling back to trust.
Open Source vs Closed source is not where I or the security professionals I know put most trust emphasis. I would enthusiastically trust something closed from Google over a rando open source project.
But back to the original point, even the most basic audit steps are the same on an open source project vs closed one. Observe what the binary does & inspect it for standard patterns.
This is rarely the case unfortunately, and for most of open source prebuilt software you use, you rely on trust and not on audit.