Hacker News new | ask | show | jobs
by jcrites 3037 days ago
It seems like Nix could benefit from a first-class concept for pinning the channel and managing updates to the channel. To achieve build reproducibility, the default outcome of checking out a project and building it should be to get the same result as everyone else who does so at the same project revision. I'd expect there to be a way to take a snapshot of the channel and lock those versions in the project until you're ready to upgrade.

I have tinkered a lot with Nix, but have not used it for any production system. I'd be reluctant to use Nix in production if there wasn't a good way to pin the channel, so that everyone who is developing on a project sees precisely the same versions as each other, and everyone who is patching a specific version of a project sees the same version as what's currently deployed. In my opinion, this should be the default behavior and not something that you have to go out of your way to achieve.

Maybe you could model this as something like a "channel fork" or "user channel". When you create a new project, you also create a channel for it that inherits from an existing channel, except that the versions of all packages are pinned and you can control the way in which changes make their way into your user channel from the upstream.

My company has a proprietary technology that's similar to Nix, and we tackle the build reproducibility problem in this way, by making it easy to branch off the main channel and control how updates make their way into it (which is like a merge). The default behavior when creating a project is to also create a project-specific channel, derived from the upstream, company-wide channel. Dependencies are declared at the major version level only, and versioning beneath that level happens implicitly via the project channel.

In this system you think of all channels as having revisions. Developers can control how their channel merges in changes from its upstream channel, which provides stability and reproducibility: until you merge and release a new version, `my-channel` always references exactly the same package versions for all developers. The default behavior of channels is to merge new versions from the upstream channel periodically, but developers can decide what's right for them, whether staying on the bleeding edge or prizing stability.

Software builds, releases, and deployments always happen in the context of a specific channel revision (`my-channel@revision`), which can be named and referenced, so if necessary you can inspect or replicate exactly the code that's part of a colleague's configuration or a production system. Not just approximately but exactly, for every package involved down to `gcc` and `glibc`. It's easy to create a workspace with the package versions specified by `my-channel@12345`.

By further tracking which source code commit hash corresponds to each package release in the channel, you can trivially look up the exact source of all packages (my-channel@12345 -> FooPackage@a1b2c3d4). Extremely useful! When you work and version things in this way, you hardly need version numbers beneath the major version level.

I suppose it's reasonable that the Nix project doesn't want to host binary versions of lots of old packages to make this use-case fast. In that case, I'd want it to be easy to clone the binary version of those dependencies and source them locally. Maybe this could operate as something like a caching proxy in front of Nixpkgs, that will store my own permanent copy of any package that I access or build. On the other hand, keeping builds forever is expensive, so perhaps this is an opportunity for Nix to provide a commercial offering with private channel hosting and a package cache with infinite TTL.

2 comments

From a technological standpoint, it's easy to pin the channel. It's as simple as creating a file like this in your project: https://github.com/catern/nix-utils/blob/master/pkgs.nix

And importing that file whenever you want to use packages from nixpkgs: https://github.com/catern/nix-utils/blob/master/default.nix

If you want to have a separate channel which you can update on your own, just fork nixpkgs on Github and point your pkgs.nix at your fork.

The Nix binary cache keeps every version of every binary in nixpkgs that's ever been built on the Nix project's build servers. (Note that that's not absolutely everything in nixpkgs, since some things aren't built centrally on the build servers for various reasons, but it's most things.)

However, I might be misunderstanding. Was there some other feature or ergonomics benefit that you were looking for?

> I suppose it's reasonable that the Nix project doesn't want to host binary versions of lots of old packages to make this use-case fast. In that case, I'd want it to be easy to clone the binary version of those dependencies and source them locally. Maybe this could operate as something like a caching proxy in front of Nixpkgs, that will store my own permanent copy of any package that I access or build.

I don't know when it will be ready, but some work has gone into backing Nix caches with IPFS, and it looks like it could be a really cool solution for making it easy to share a cache for 'extra' or 'old' packages without much infrastructure work on the parts of users.

https://github.com/NixOS/nix/issues/859