[sgammon]

Sharing with team members, sharing with CI, and the ability to pull from more than just what's on your machine (i.e. a larger addressable cache than you are willing to keep on disk). Cache objects also compound across projects, so it's nice to ship them up somewhere and have them nearby when you need them.

Re/spoofing, obviously it's all protected with API keys and tokens, and we're working on mechanisms to perform end-to-end encryption. In general, build cache objects are usually addressed by a content-addressable-hash, so that also helps because your build typically knows the content it's looking for and can verify.

That isn't true for all tools, though, so we're working to understand where the gaps are and fix them.

[xjia]

IIUC the actual computation (e.g. compiling, linking, ...) happens on client (CI or developer) machines and the results are written to the server-side cache.

By spoofing I meant to say that an authenticated but malicious client (intentionally or not, e.g. a clueless intern) may be able to write malicious contents to the cache. For example, their build toolchain could be contaminated and the resulting build outputs are contaminated. The "action" per se and its hash is still legit, but the hash is only used as the lookup key -- their corresponding value is "spoofed."

The only safe way I can imagine to use such a remote cache is for CI to publish its build results so that they could be reused by developers. The direction from developers to developers or even to CI seems difficult to handle and has less value. But I might be missing some important insights here so my conclusion could be wrong.

But if that's the case, is the most valuable use case to just configure the CI to read from / write to the remote cache, and developers to only read from the remote cache? And given such an assumption, is it much easier to design/implememt a remote cache product?

[sgammon]

All great points but in practice, tools like Bazel and sccache are incredibly conservative about hashes matching, to include file path on disk and even env var state.

One goal of these tools is to guarantee that such misconfiguration results in a cache key mismatch, rather than a hit and a bug.

There are tons of challenges designing a remote build cache product, like anything, but that one has turned out to be a reliable truth.

Some other interesting insights:

- transmitting large objects is often not profitable, so we found that setting reasonable caps on what’s shared with the cache can be really effective for keeping transmissions small and hits fast

- deferring uploads is important because you can’t penalize individual devs for contributing to the cache, and not everybody has a fast upload link. making this part smooth is important so that everyone can benefit from every compile.

- build caching is ancient, Make does its own simple form of build caching, but the protocols for it vary in robustness greatly, from WebDAV in ccache to Bazel’s gRPC interface

- most GitHub Actions builds occur in a small physical area, so accelerating build artifacts is an easier problem than, say, full blown CDN serving

The assumptions that definitely help:

- it’s a cache, not a database; things can be missing, it doesn’t need strong consistency

- replication lag is okay because a build cache entry is typically not requested multiple times in a short window of time; the client that created it has it locally

- it’s much better to give a fast miss than a slow hit, since the compiler is quite fast

- it’s much better to give a fast miss than an error. You can NEVER break a build; at worst it should just not be accelerated.

It’s an interesting problem to work on for sure.

[Thorrez]

>In general, build cache objects are usually addressed by a content-addressable-hash

How does that work? I would think the simplest case of a build object that needs to be cached is a .o file created from a .c file. The compiler sees the .c file and can determine its hash, but how can the compiler determine the hash of the .o file to know what to look up in the cache? I think the compiler would need to perform the lookup using the hash of the .c file, which isn't a hash of the data in the cache.

[aseipp]

In the case of the Remote Execution/Cache API used by Bazel among others[1] at least, it's a bit more detailed. There's an "ActionCache" and an actual content-addressed cache that just stores blobs ("ContentAddressableStorage"). When you run a `gcc -O2 foo.c -o foo.o` command (locally or remotely; doesn't matter), you upload an "Action" into the action cache, which basically said "This command was run. As a result it had this stderr, stdout, error code, and these input files read and output files written." The input and output files are referenced by the hash of their contents, in this case, and they get uploaded into the CAS system.

Most importantly you can look up an action in the ActionCache without actually running it, provided you have the inputs at hand. So now when another person comes by and runs the same build command, they say "Has this Action, with these inputs, been run before?" and the server can say "Yes, and the output is a file identified by hash XYZ" where XYZ is the hash of foo.o, so you can just instantly download it from the CAS.

So there are a few more moving parts to make it all work. But the system really is ultimately content-addressed, for the most part.

[1] https://github.com/bazelbuild/remote-apis/blob/main/build/ba...

[bgmeister]

If you're only using remote caching (ie no remote execution) then all cache clients need to trust each other, because a malicious client can upload any result it wants to a given ActionCache key, and there's no way to verify the ActionCache entries are correct unless the actions are reproducible. (And verifying ActionCache entries by rerunning the actions kind of defeats the purpose of using a build cache.)

If you don't want clients to have to trust each other, then you can block ActionCache write access to the clients and add remote execution. In this setup clients upload an action to the CAS, remote executors run the action and then upload the result to the ActionCache, using the hash of the action as the key. This way malicious clients can't spoof cache results for other clients, because other clients won't ever look for the malicious action's key in the ActionCache.

[sgammon]

In Bazel’s case and other cases, build cache objects are held in CAS and then referenced from other keys. I believe BuildXL from Microsoft also works this way.

Of course one other advantage to build caches is they are verifiable: the intent is to produce the exact same output as a normal call, and that’s easily checked on the client side.

No question that build caching poses inherent supply chain risks though and that’s part of what we want to solve. I think people are hesitant to trust build caching for good reason until there are safer mechanisms and better cryptographic patterns applied.

[sgammon]

Yep, aseipp, and we support the full gRPC interface for remote caching offered by Bazel, including the newer APIs.

Explained better than I could for sure. I find it very interesting how BuildXL and Bazel ended up at similar models for this problem. I don’t yet know the history of which informed which.

(As compared to, say, Gradle, which works based on input hashes instead.)

[krupan]

When a .o is stored in the cache it is associated with the hash of the .c file

[sgammon]

(Fwiw, group conversation encryption tech like MLS is somewhat applicable, and that's the sort of pattern we're looking at, but it would be cool to know if that's moving to you on the problem of safety w.r.t. builds.)