[kardos]

Great article, impressive attention to detail. It sounds like a situation that would benefit from a compiler cache, such as the one found at https://ccache.samba.org/. Is there any benefit to be had there or is the storage tradeoff too expensive?

[jperkin]

I want to try ccache at some point (with pkgsrc it's trivial to add), but I'm not convinced it will help.

Given the distributed nature the first implementation would keep the ccache objects on NFS, and that's going to cause additional latency for every cc invocation to the point where even a cache hit will likely be slower.

The next step would be to synchronise all of the ccache objects back to each build zone and then loopback-mount the directory into each chroot. This may provide a small increase in performance, but at the cost of additional complexity. I also wonder whether it would be enough of a performance win compared to the extra time it will take to rsync the objects to each build zone at the start of the build.

I will likely do this at some point anyway, as I'd like to have some hard numbers to back up my theory, and if it turns out to be a win, even better ;)

[tlb]

I've built a few such configurations myself, but never again. The worst behavior is that sometimes, the NFS caches result in compiling old versions of some files some of the time. When you're rapidly tweaking things for debugging and sometimes some changes only make it into some compilation units, it can drive you insane.

And it doesn't go much faster than just compiling locally with Clang on an 8 core CPUs with SSDs.

[kardos]

You could have a separate ccache directory for each package, such that you can enable/disable using ccache on a per-package basis. Then you can rsync the objects in parallel with the build process, but only for the 'long tail' packages (kde, etc), such that the ccache objects are available by the time that the those packages start building. Definitely adds complexity though.