Each object file (.o) has a unique hash and is stored as thehash.o.
It's certainly much faster to download the .o than it is to build it. Once it's downloaded it stays on the local filesystem until it's garbage-collected.
The whole point of S3 is that it is inexpensive. You don't want to pay premium money for terabytes of data that are usually invalidated everyone someone makes a significant change.
Our blobs end up quite a bit smaller than that, and I can see using S3 for many build cache problems. In our case in-memory worked better with overflow to disk for large or infrequently used objects.
R2 is S3 to me, btw, or at least it’s the same API. Object storage as a model is really what I’m asking about. I’m genuinely glad to hear S3 was enough to be impactful.
That depends on your network latency and throughput, plus the performance of your compiler.
With fast compilers and less bloat (not C++ nor Rust) such a distcc is certainly slower.