The threat is not that the cache contains builds of untrusted code but that it contains builds that do not match the code that they are associated with.
It uses a cryptographic hash of the _inputs_ to the compiler, but there is no way to verify that the cached artifact matches the _output_ of the compiler without actually compiling it yourself.