The threat is not that the cache contains builds of untrusted code but that it contains builds that do not match the code that they are associated with.
It uses a cryptographic hash of the _inputs_ to the compiler, but there is no way to verify that the cached artifact matches the _output_ of the compiler without actually compiling it yourself.
Let’s say you have 15 engineers and they each have their own laptop computer. Each of these engineers generates a pair of cryptographic keys, one public and one private.
Each engineer then gives their public key to the trusted authority that operates the ccache server. Only code that is submitted and signed by a respective private key is built and then distributed to the rest of the engineers.
For a public project you would only want the builds to be propagated out to other developers once the changes had been approved and then merged into a branch that triggers the CI.