[neuromantik8086]

Guix is one of several solutions that has been touted as a solution. Another one that is quite popular in HPC circles is Spack (https://spack.readthedocs.io/en/latest/).

At my institute, we actually tried out Spack for a little bit, but consistently felt like it was implemented more as a research project rather than something that was production-level and maintainable. In large part, this was due to the dependency resolver, which attempts to tackle some very interesting CS problems I gather (although this is a bit above me at the moment; these problems are discussed in detail at https://extremecomputingtraining.anl.gov//files/2018/08/ATPE...), but which produces radically different dependency graphs when invoked with the same command across different versions of Spack.

I've since come to regard Spack as the kind of package manager that science deserves, with conda being the more pragmatic / maintainable package manager that we get instead . Spack/Guix/nix are the best solution in theory, but they come with a host of other problems that made them less desirable.

[jnxx]

> Spack/Guix/nix are the best solution in theory, but they come with a host of other problems that made them less desirable.

I would be quite interested to learn more what these problems are, in your experience. I've only tried Guix (on top of Debian and Arch) and while it is definitively more resource-hungry (especially in terms of disk space), I don't percive it as impractical.

[yjftsjthsd-h]

As someone coming from the computing side of things, I found nix to be quite difficult to grok enough to write a package spec, and guix was pretty close, at least in part because of the whole "packages are just side-effects of a functional programming language" idea. At least nix also suffers from a lot of "magic"; if you're trying to package, say, an autotools package then the work's done for you - and that's great, right up until you try to package something that doesn't fit into the existing patterns and you're in for a world of hurt.

Basically, the learning curve is nearly vertical.

[rekado]

> guix was pretty close, at least in part because of the whole "packages are just side-effects of a functional programming language" idea

This must be a misunderstanding. One of the big visible differences of Guix compared to Nix is that packages are first-class values.

[yjftsjthsd-h]

You're right; on further reading I can see guix making packages the actual output of functions. I do maintain that the use of a whole functional language to build packages raises the barrier to entry, but my precise criticism was incorrect.

[neuromantik8086]

I can only speak to Spack in particular, but the main issue that I found with it was balancing researcher expectations for package installation speed with compile times. For most packages, compile times aren't a huge problem, but compilers themselves can take days to build, and it isn't unheard of for researchers to want a recent version of gcc for some of their environments.

In theory this isn't an issue with Spack (assuming that you have a largely homogeneous set of hardware or don't use CPU family-specific instruction sets), since you can set up cached, pre-compiled binaries on a mirror server (similar to a yum repo) and have people install from there.

Spack, however, has a lot of power/complexity. A lot of untamed power that means that bugs can sometimes be more likely than in other, more mature (or mature-ish) package managers. Namely, Spack allows you to not only specify the version number of a package, but also the compiler that you use to make that package, specific versions of dependencies that you want to use, which implementation of an API you want to use (i.e., MPICH or OpenMPI for MPI), and compiler flags for that package. When you run an install command / specify what you want to install, Spack then performs dependency resolution and "concretizes" a DAG that fulfills all of the constraints.

The issue that I ran into was that if you don't specify everything, Spack makes decisions for you about which version of a dependency, which compiler, etc to use (i.e., it fills in free variables in a space with a lot of dimensions). This would be great and dandy normally, although the version of Spack that I used occasionally constructed totally different graphs for the same "spack install gcc" command (if I recall correctly; take all of this with a grain of salt b/c I might be misremembering). This meant that it wouldn't use cached versions of gcc that had already been built, and ended up rebuilding minor variants of gcc with options I didn't care about.

At National Labs and larger outfits, the trade-offs between this kind of complexity/power and the accompanying bugginess (Spack has yet to hit 1.0) seem to favor complexity/power while accepting these sorts of bugs, but I don't work at a larger outfit and my group didn't need that level of power/control over dependencies and rather needed something that "just worked" and would allow researchers to be able to install packages independently of us (IT people). conda (mostly) fit the bill for this. I still think that Spack is the future and it has a special place in my heart, but it will have to be more stable for me to want to use it in production.