It isn't possible, in most cases, to trust open source software either. Have you verified that the binaries on your phone were indeed built from the source you can read on github or wherever?
Which fully open-source phone platform do you have in mind? I'm not aware of any.
On desktop and servers, however, it certainly is possible (and not-too-impractical) to verify binary blobs against known PGP signatures. See Debian's reproducible builds, for instance.
On desktop and servers, however, it certainly is possible (and not-too-impractical) to verify binary blobs against known PGP signatures. See Debian's reproducible builds, for instance.