[aseipp]

Not just team members; if you make your cache publicly readable, contributors to e.g. your GitHub/GitLab/Whatever project can also use them and get really fast builds, the first time they try to contribute. So a remote cache is nice to have, if it's seamless.

Nix works this way by default (and much of the community operates caches like this) and it can be a massive, massive time saver.

> How do you make sure the build results are not spoofed?

What do you mean "spoofed?" As in, someone put an evil artifact in the cache? Or overwrote an existing artifact with a new one? Or someone just stole your developers access and started shoving shit in there? There's a whole bunch of small details here that really matter to understand what security/integrity properties you want the cache to uphold.

FWIW, I've been looking into this in Buck2/Bazel land, and my understanding is that most large orgs just use some kind of terminating auth proxy that the underlying connection/flow/build artifacts can be correlated back to. So you know this cache artifact was first inserted by build B, done by user X, who authenticated with their key K, etc etc.

[sgammon]

Exactly — just like Git, everything is ultimately identified with a key which can tie back to a stable identity thru OIDC or similar mechanisms. At least that’s how we did it.

[yjftsjthsd-h]

Nix only caches at the package level, doesn't it?

[sgammon]

Nix is different, yeah, and it won’t wire together a build cache for you. Nix is great for many things of course, it’s just not a replacement for sccache per se

Nix + sccache would probably be pretty great for preserving paths and environment, which is really healthy for build caching in general.

[mikepurvis]

Properly handling cargo and bazel builds in nix is very much not a solved problem, and Nix's resistance to allowing ccache into the sandbox for purity reasons definitely exacerbates the problem.

[aseipp]

sccache should not be allowed in the sandbox, at least not just bolted on; realistically such cases are probably better handled in the long run by Recursive Nix, so that you can build new derivations inside of existing ones, and the results are cached in the same (outer) store. This means there won't be duplicate caches; /nix/store and where-ever sccache puts results.

Cargo is a good example. For a number of (practical) reasons, cargo -> nix translators are a lot of effort and often have bugs, so for "simplicity" all upstream crates just compile every crate dependency every time. That means if two crates use the same dependency, it gets compiled twice. It's important to understand this is no worse than the way Cargo works already for most people. Cargo does not have a content address storage model, unfortunately. But it's pretty annoying for Nix users and costs a lot.

In theory, we could wrap rustc with a recursive-nix enabled wrapper so that rlib's etc each get a granular derivation and then get put into the host store. So assuming a crate gets built with the same flags, between two "outer" expressions, they'll get to share the work and it will go into the store. A working example of this for C++ code is nix-ccache, but a fully robust implementation is a bit of work.[1]

Recursive Nix is still experimental but there is some use of it (privately and publicly) that I know of for these purposes.

[1] https://github.com/edolstra/nix-ccache

[sgammon]

It sounds like you’re really experienced with Nix or perhaps a contributor. We’re new to Nix at Elide (makers of Buildless) but we’d love to collaborate :)

I just tried it for the first time the other day and although I’m not ready to move to it yet, I can already see the brilliance.

[mikepurvis]

As a user (and occasional recompiler) of the tensorflow derivation in NixOS, I'd love to see Nix able to somehow do a poetry2nix style transformation on bazel dependencies so they would be properly cached in individual store paths.

[aseipp]

It can cache at the level of any set of files, technically speaking (a single file, directory of files, and so on); so you could even use it as a Makefile replacement or something for example. But most people don't do that; the ecosystem is broadly much more coarse-grained and designed around "packages", yes.

I wasn't really referring to the coarseness though; just that a lot of Nix projects provide build caches to speed things up for contributors. It's not just something for internal teams. And it really does help.