[pxc]

I'm in the process of rolling out something analogous at work, where Nix jobs run inside rootless Podman containers but the Nix store and Nix daemon socket are passed through from the host, so the jobs' dependencies all persist, dependencies shared between projects are stored only once, when two concurrent jobs ask for the same dependency they both just wait for the dependency to be fetched once, etc.

We also currently have some jobs that build OCI images via the Docker/Podman CLI amd build using traditional Dockerfile/Containerfile scripts. For now those are centralized and run on just one host, on bare metal. I'd like to get those working via rootless Docker-in-Docker/Podman-in-Podman, but one thing that will be a little annoying with that is that we won't have any persistent caching at the Docker/Podman layer anymore. I suppose we'll end up using something like what's in the article to get that cache persistence back.

[adityamaru]

> Nix jobs run inside rootless Podman containers but the Nix store and Nix daemon socket are passed through from the host

That's a neat idea, was the primary motivation for building this out the perf gains on the table?

[pxc]

More or less! Before that we basically had two kinds of jobs: one where Docker provided the job's dependencies and one where Nix provided the job's dependencies. The former was, naturally, always containerized, but the latter ran on bare metal. (We also had some jobs that used Nix without a persistent /nix/store, but never mind that.) Given both options, choosing Nix over Docker is really nice in two ways: (1) you don't need to deal with any infrastructure (i.e., push an image somewhere) for your job; and (2) even so, the second run and later can be really, really fast. We also use Nix to provide dependencies on our local machines for some projects, so just reusing that same environment in CI is a natural fit, too.

But as we started to mature our own CI 'infrastructure' (the automation we use to set up our self-hosted runners), I wanted to containerize the Nix builds. Using 'shell executors' in GitLab just feels icky to me, like a step backwards into Jenkins hell. Those jobs do leave a little bit more behind on disk. More importantly, though, while all of my team's Nix jobs use Nix in an ephemeral way, it is possible to run `nix profile install ...` in one of these bare metal jobs. That could potentially affect other such jobs, plus it creates a 'garbage collector root' that will reduce how much `nix-collect-garbage` can clean up a little bit. Our jobs are ones we'd like other teams across the company to run, and so we also want to provide some really low-effort ways for them to do so, namely: via shared infrastructure we host, via any Docker-capable runners they might already have, and by leveraging the same IaC we use to stand up our own runners.

To that end, we really want to have just one type of job that requires just one type of execution environment, and we definitely want opt-in persistence instead of a mess where jobs can very easily influence one another by accident or malice. But we don't want to lose the speedup! The real action in these jobs is small, so by sharing a persistent Nix store between runs, they go down from 2-10 minutes to 2-10 seconds, which is the kind of UX we want for our internal customers.

The new Nix image is more suitable for all three target scenarios: it's less risky on runner hosts shared by multiple teams, it still works normally (downloading deps via Nix on every run) on 'naive' Docker/Podman setups, and our runner initialization script actually uses Nix to provide Docker and Podman (both rootless), so any team can use it on top of whatever VM images they're already using for their CI runners regardless of distro or version once they're ready to opt into that performance optimization.