[dataflow]

"Just files" doesn't quite capture the complexity of the situation to me though. Say I happen to install package X via apt and Y via nix, and both of them depend on Z (in apt and nix respectively), and Z needs to bind to a port, then I imagine both will install but one of them will break, possibly including their dependents. Or if I install a package on Nix that expects a certain syscall that's not in the Ubuntu kernel yet (maybe like the recent /usr/bin/sleep issue with WSL), then that either breaks Nix or my ability to keep using an Ubuntu kernel. Right? And there are probably other things I'm just lacking the imagination for right now. But it seems to me there are all sorts of conflicts that can come up in practice. I've seen enough trainwrecks when upgrading even across OS versions that I have a hard time seeing how running 2 package managers can work on a single OS without breaking something?

[brmgb]

> "Just files" doesn't quite capture the complexity of the situation to me though.

I think that's the root of the issue.

You seem to be somewhat confused about how Linux executes programs. An executable is just a file on the disk. It might be entirely self-sufficient or it might use libraries. Libraries are just files on the disk too (they end in .so.number where number is a version number). To load the libraries, when it starts a program, Linux first runs another executable called a dynamic linker (usually ld). Where the dynamic linker looks for the libraries files depends of its configuration. Ld will notably look into the paths listed in the LD_LIBRARY_PATH variable.

All of this has nothing to do with distributions. It's just how Linux works and will always be true. As long as you have all the so your executable need in a place where ld can find them, it will run.

Now, what a package manager does when you install a package is just putting all the files contained within it in the correct location, ensuring you will have installed the necessary shared libraries for the executable you want to run and setting up everything that might need to be setup.

You can manually add executables and so files as much as you like. The only issue you might encounter is using a location which is also used by a package. Depending of the package manager you use, that might either overwrite your modification or make the package installation fail.

While I have never used it, apparently Nix is well behaved and only install things into /nix so that shouldn't be an issue.

[takeda]

> Say I happen to install package X via apt and Y via nix, and both of them depend on Z (in apt and nix respectively)

That actually is impossible, Nix will only depend on packages in nix, and nothing else. So whatever you have installed won't affect it.

The important part of Nix on Linux is patchelf[1] basically binaries generated are processed by it, this rewrites elfs to link to libraries in the /nix/store

Regarding syscalls, if you use NixOS then you're tied to specific state of nixpkgs, which also dictates the kernel installed. So you shouldn't run into it. You probably might run if you install Nix on Ubuntu. I don't remember running into it, and I think it should be rare since linux ABI supposed to not break compatibility.

[1] https://github.com/NixOS/patchelf

[dataflow]

>> Say I happen to install package X via apt and Y via nix, and both of them depend on Z (in apt and nix respectively)

> That actually is impossible, Nix will only depend on packages in nix, and nothing else. So whatever you have installed won't affect it.

You misunderstood the scenario. X, Y, Z are package names here. Like Z might be openssh, and it might have dependent X in apt, and Y in nix. You'd get an apt installation of Z and a nix installation of Z.

[geofft]

> Say I happen to install package X via apt and Y via nix, and both of them depend on Z (in apt and nix respectively), and Z needs to bind to a port, then I imagine both will install but one of them will break.

When you install a package with apt, rpm, etc., it has the ability to run code at install time, create init scripts/systemd unit files, etc. When you install a package with Nix, it does not have that ability. So if you install Y via Nix, it will install Z via Nix, and Nix's Z will not be started automatically. You'll need to start it yourself (or Y will need to trigger it, when you run it, etc.)

If you also install X via apt and Z via apt, apt's Z will be started automatically, and will prevent Nix's Z from starting at the same time. You'll need to stop apt's Z.

Note that there's a spectrum here - Red Hat-based distros conventionally don't start software automatically when you install it, although they do leave the configuration in place for you to enable it (with chkconfig or something). Debian-based distros do. (While I am generally a Debian fan, this is one of the things I don't like about Debian ... possibly because I lost some mail once when I did "apt-get install exim" on a production server that had died, and a bunch of email that had been in other people's queues hit the default exim and then got bounces because I hadn't configured it yet.)

Another way of thinking about it is that there are two meanings of "install GRUB". One is that the GRUB commands are available for you to use. One is that your disk has GRUB in its boot sector. Arch, Debian, Red Hat, etc. all interpret "install GRUB" as doing both of these, but doing just the first one is valid, and then it's up to you to use the GRUB commands to put GRUB on the boot sector of whatever disk you want.

> Or if I install a package on Nix that expects a certain syscall that's not in the Ubuntu kernel yet (maybe like the recent /usr/bin/sleep issue with WSL), then that either breaks Nix or my ability to keep using an Ubuntu kernel. Right?

Yes. Nix won't install a new kernel for you, unless you're using NixOS.

(Technically, I suppose you can put a new kernel down on disk with Nix, but it won't install it in the sense of changing systemwide configuration.)

> I've seen enough trainwrecks when upgrading even across OS versions that I have a hard time seeing how running 2 package managers can work on a single OS without breaking something?

This is generally due to one of two things, both of which Nix sidesteps:

- There are incompatible updates to a file, e.g., you install "mawk," which provides /usr/bin/awk, but some other package expected that to be "gawk" and uses gawk-specific features. Nix doesn't have an equivalent to /usr/bin; there are no paths that are shared across packages. Each file is in a directory which corresponds to the identity of the one package (at one specific version) that provides it, and other packages bake in those dependencies. This is the fundamental cool thing about Nix.

- Systemwide changes when you install/remove stuff, like uninstalling syslogd shutting down your syslog daemon, installing a kernel changing the default kernel you'll boot into, etc. Installing a package in Nix doesn't have any effects beyond putting down files. The most Nix will do is keep a pointer (a "profile") to some set of stuff you're interested in (again, by exact identity). You can change that pointer, but there's nothing special about your particular pointer. If you point to a set of fewer things, other people can keep pointing to the things you no longer see. If you point to more things, that doesn't cause any code to run automatically.

Combined, this means that Nix stays out of the way of a traditional package manager. (Also, it means that you can set up your system so non-root users can install/remove things with Nix - you need to do some setup as root tp install Nix in the first place, but once it's there, users can't mess with each other.)

If you're familiar with virtualenvs etc., you can sort of think of it like that (although it's not a perfect metaphor) - you can install Python packages into a specific environment, but they don't get installed systemwide, and if you install, say, a web server (gunicorn, Flask, Django, etc.) into multiple virtualenvs, none of them are started automatically. You can't meaningfully install a kernel into a virtualenv (nothing stops someone from releasing a kernel on PyPI, but "installing" it will just download the file and put it somewhere and nothing else).

This does mean that the experience of using Nix is different from using a traditional package manager - you'll need to take care of starting the services you need, etc. Basically Nix reduces the scope of a package manager to say that running services isn't part of it, and then it does a better job of implementing that reduced scope. If you're a company like Shopify that is unlikely to be happy with the built-in web server that automatically starts when they "apt-get install apache2" that shows "Congratulations, you've installed a web server" to your customers, that's totally fine, you were going to do your own service management anyway.

[mehrdadn]

Thank you! Your comments are always great. :) Especially this bit was I think the insight I was looking for:

> This does mean that the experience of using Nix is different from using a traditional package manager - you'll need to take care of starting the services you need, etc.

I was trying to figure out where exactly the trade-offs would be -- so that's one of them: it seems like (in some sense, to simplify) Nix takes responsibility for the data & programs, and leaves you with the responsibility of handling the control flow (at least externally).

If you don't mind, this leaves me with one more question. A typical headache that comes up during upgrades (on pretty much any distro) is config changes -- i.e., where the user and the package manager are both responsible for handling data. (Contrast this with the data/control-flow responsibility split we discussed above.) Like you modify /etc/ssh/sshd_config or any other one of the countless /etc config files that you can't configure with a .d folder, then the system upgrades and leaves you to handle the merge conflict. How does Nix deal with these? Presumably "never modify /nix/etc/..." is neither a realistic expectation for the user nor for the package manager, so there are bound to be merge conflicts, right? But that goes squarely against having a deterministic build configuration, so what happens?

[geofft]

So, there is no /nix/etc - anything in /nix is immutable. I haven't done this and hopefully someone will correct me if I get the details wrong, but I think the way you'd make this work within nix is you'd make your own package, let's call it "my-sshd-config," and it depends on a certain version of sshd. Then you'd have /nix/store/abcd1234-my-sshd-config-1.0/etc/sshd_config. (where "abcd1234" is a hash of everything in your package, including the specific version of ssh you depend on)

If you want to upgrade sshd and the config changes, you'd make my-sshd-config 2.0, and nix would put its files in /nix/store/efgh5678-my-sshd-config-2.0/etc/ssh_config. You could have both of these installed at the same time.

Then, it's up to you to stop the sshd running out of my-sshd-config 1.0 and to start the one running out of my-sshd-config 2.0. Once you're confident of the upgrade, you can then tell Nix to clean up my-sshd-config 1.0, but that's just removing files, since you've already stopped the service.

(We're about to upgrade sshd at work next week and I wish we had something like this, honestly, both so I could distribute the files to machines well before the upgrade and so that it's easy to roll the upgrade back - just stop the new sshd and start the old one. Then there wouldn't be concerns about config drift, forgetting to revert files, etc. The only configuration that I'd have to manage is which one is active.)

Most people who use Nix without NixOS aren't running services like sshd out of it, but NixOS will do basically this for handling service upgrades. Conceptually NixOS gives you one mutable configuration knob, which is "what is all the systemwide stuff for this system." You can depend on some particular sshd and its config, some particular init system and its config, some particular syslogd and its config, etc. If you want to change any config, you make a new package with the changed config, and then flip the one mutable thing to point to it instead. (This does mean that it's easy to roll back your system to an old state if you realize you made a mistake!)

There is a limitation here, which is that Nix can't do a lot about config in your home directory, like ~/.ssh/ssh_config (for the SSH client). You'll have to be careful with that just like you would with upgrades in general. Alternatively, you could make your own Nix packages that include client config, and avoid storing config in your home directory.

[mehrdadn]

Oh dear... so you have to write a Nix package (and learn how the language and package management work) just to modify global config files, instead of just editing them? That sounds like an absolute nightmare for a local machine, though possibly a great tool for automated systems.

Also, if there's no /nix/etc... then what is Nix modifying? sshd (or any other more common program; I'm just using sshd as an example to understand the rest of the system) won't magically know to look at /nix/store/efgh5678-my-sshd-config-2.0/etc/ssh_config, right? I thought you'd at least need a symlink like /nix/etc/ssh/sshd_config to link to it. Unless you're saying Nix modifies your system's /etc/ssh/sshd_config directly and makes it a symlink to the immutable Nix store? In which case, wouldn't it just trample everything your own distro's package manager does in that folder?

[singron]

NixOs manages configuration in a really elegant way with modules. Modules are a structured way of combining different sources of configuration.

E.g. you might need to configure a list of users on the system. You might manually configure your user in your top-level module:

  users = [ "dataflow" ];

But if you have postgres enabled, then it might configure a user in it's own module.

  users = [ "postgres" ];

When the modules are evaluated and combined, you can end up with a list that contains both:

  users = [ "postgres" "dataflow" ];

Compare this to conventional distros. If you edit your distro's default config file, and the distro updates the default config file, how will your modified config be updated? E.g. Arch just puts the new config in a .pacnew file, and you will just keep using your old config. pacman prints a line out, but it often scrolls off the top of the screen and it's easy to miss. It's up to you to manually merge those configs now. E.g. my arch system has 22 .pacnew files in /etc, and occasionally things break or don't work as well because updates aren't applied to the configs. Debian has debconf, but that always seemed more like a set of ad-hoc workarounds than a solution.

[takeda]

It's not done that way.

If you install Nix on Linux or OS X, you just have packages installed but configuration (if any) you have to edit manually like you would in another Linux distro.

If you use NixOS in addition to packages it also uses modules. Modules take care of setting up services, adding users/groups etc and placing the configuration in right place. You use module configuration to configure the application, and Nix generates actual config.

Here[1] is example of how PostgreSQL is configured.

[1] https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/s...

[GlennS]

I think most people who use Nix on Redhat, Debian, Ubuntu and so on probably don't use it to provide system services like `sshd`.

A more common use would be to install a package in a version that isn't available, or where you would need to link to a different version of the library than is available on your system. Nix makes the dreaded autotools dance much more bearable, even for someone like me who doesn't really know any C++.

Another common situation is when you are a software developer working on many projects with overlapping but slightly different dependencies where slightly different sometimes turns out to be incompatible. Ideally you could handle this with `virtualenv` or your language's equivalent, but I often find I end up with multi-language projects where this just doesn't work. For example, if one of my Python libraries also wants some C libraries in a particular version. Nix is ideal for such a situation.

Docker and docker-compose can work too, but, eh, it's kind of slow, and it'll probably break a few months down the line when all the JavaScript library authors decide that building things with `plink` is gauche and everyone should move to `smooch` instead, or when CRAN decides it doesn't want to keep these old versions of libraries any more.

And, what do you do about your linting and formatting and code completion tools? Do those go inside Docker too, because then the Docker slowness really shows up.

(If you don't have these sorts of problems often, then Nix may not help you all that much. I guess reducing the amount of stuff you install and uninstall with `apt` or `rpm` probably still reduces the rate at which your system rots though.)

Having said all that, if you decided that you did want to use Nix to provide packages for your system services, then there's no difficulty in specifying a mutable config file in /etc/wherever. That part would be handled in your init system config (say, a systemd unit) instead of inside Nix.

It's for NixOS that you would use the Nix language to configure everything, and I've found that experience to vary quite a lot.

I've used some obscure or packages which didn't expose enough of the configuration to do what I needed by default, and so I had to make my own version. This can be pretty annoying and time consuming.

Nixpkgs' maintainers get a lot of work done individually because of the tooling, but there just aren't enough of them yet.

On the other hand, I've found that configuring popular things like Nginx and Postgres is often much easier in the Nix language.

It's much harder to make a small syntax error, you have access to some helpful templating constructs, and you can re-use the same constants for different config files in different packages.

Similarly, for NixOS, configuring your systemd services is really well handled.

Wow, that ended up long winded. Sorry.

[tomberek]

Interestingly, sshd WILL magically know to look at /nix/store/efgh5678-my-sshd-config-2.0/etc/ssh_config. Usually you would use sshd as a systemd service (possible to do and manage with Nix in Ubuntu, but why would you?). If so, you would write the systemd configuration using Nix and the service file deployed to NixOS (or Ubuntu, imagine deploying some custom/proprietary service you want isolated from whatever else the client has on the machine) would have all the long hashes auto-magically inserted to produce this (this is on my machine, all i had to write was "services.openssh.enable = true;" but additional configuration is available, see https://nixos.org/nixos/options.html#services.openssh.):

  [Unit]
  After=network.target
  Description=SSH Daemon
  X-Restart-Triggers=/nix/store/d4ys2c8kzzcp3g4fv3ivy7a5nkayg7w2-sshd.conf-validated

  [Service]
  Environment="LD_LIBRARY_PATH=/nix/store/71mr6yjmia7y8lw4g5ghk5ag9yq5ir2i-nss-mdns-0.10/lib:/nix/store/zbxfs37qjj6ddrfnzrdnxnkrvvm1ddsf-systemd-245.3/lib"
  Environment="LOCALE_ARCHIVE=/nix/store/9b725cly2a6a61vb8bgz7cyr0xr8y2av-glibc-locales-2.30/lib/locale/locale-archive"
  Environment="PATH=/nix/store/5yx7mv7md9c9nldj69inrnr7rjdkzqq3-openssh-8.2p1/bin:/nix/store/miwvn81sgbbcq5bfglr6v3pwchgsd00c-gawk-5.0.1/bin:/nix/store/ca9mkrf8sa8md8pv61jslhcnfk9mmg4p-coreutils-8.31/bin:/nix/store/hg3albf7g05ljfqrfjhd58rblimrp6ph-findutils-4.7.0/bin:/nix/store/8pajzfyqx1v7dz1znrnrc4pqj5rmnx24-gnugrep-3.4/bin:/nix/store/jpqlmf3wqg281j8fdz50kjl525pfsxjc-gnused-4.8/bin:/nix/store/zbxfs37qjj6ddrfnzrdnxnkrvvm1ddsf-systemd-245.3/bin:/nix/store/5yx7mv7md9c9nldj69inrnr7rjdkzqq3-openssh-8.2p1/sbin:/nix/store/miwvn81sgbbcq5bfglr6v3pwchgsd00c-gawk-5.0.1/sbin:/nix/store/ca9mkrf8sa8md8pv61jslhcnfk9mmg4p-coreutils-8.31/sbin:/nix/store/hg3albf7g05ljfqrfjhd58rblimrp6ph-findutils-4.7.0/sbin:/nix/store/8pajzfyqx1v7dz1znrnrc4pqj5rmnx24-gnugrep-3.4/sbin:/nix/store/jpqlmf3wqg281j8fdz50kjl525pfsxjc-gnused-4.8/sbin:/nix/store/zbxfs37qjj6ddrfnzrdnxnkrvvm1ddsf-systemd-245.3/sbin"
  Environment="TZDIR=/nix/store/wmry9mqmimq8ib8ijli4g1yx92gxjli5-tzdata-2019c/share/zoneinfo"
  
  
  X-StopIfChanged=false
  ExecStart=/nix/store/5yx7mv7md9c9nldj69inrnr7rjdkzqq3-openssh-8.2p1/bin/sshd -f /etc/ssh/sshd_config
  ExecStartPre=/nix/store/1mzzy0dwjzy6kcwad7q79pvc444yn288-unit-script-sshd-pre-start
  KillMode=process
  Restart=always
  Type=simple

No symlinks to /etc/sshd. This service would be independent from other software on the host system, other than PID1 managing it.

[mehrdadn]

Interesting, thank you! So that means programs are sometimes patched to look for configs in nonstandard locations generated by Nix during installation time. But then if I wish to change any of those configs (maybe to change one of the defaults)... I have to copy them, make my modifications, generate my own package for them, and install them to wire them in as substitutes for the existing packages. Then when the upstream package changes the config file, I have to generate a new package with all the conflicts manually resolved, right? It seems a bit of an arduous process, though I do see the appeal.

[geofft]

Woah, I'm lost here. The idea is that it uses /etc/ssh/sshd_config as its input? (How do you handle upgrades, then?) What is d4ys2c8kzzcp3g4fv3ivy7a5nkayg7w2?

[geofft]

I haven't used NixOS, so take me with a grain of salt - it looks like the actual way of doing this in NixOS is that you have a systemwide configuration file that you can edit, and running "nixos-rebuild" will pick up your changes and automatically make the packages you need. See "Changing the Configuration" in the manual: https://nixos.org/nixos/manual/index.html So, at the end of the day, there is a Nix package, but you don't interact with it by using the packaging tools, you interact with it by editing a file and then running a command that snapshots the current version of the file and does everything for you.

If you're running your own services, you don't have to go through Nix packaging, you can handle this yourself if you have a way you prefer. For example, if you're running WordPress out of your home directory, you can have a git repo with some config files and a script that runs a particular version of Apache, MySQL, PHP, WordPress, etc. out of Nix. If you want to upgrade, edit the versions in the script and also the config files, then tell them all to restart. You can't rely on having a single systemwide version of Apache like you can with a traditional distro, but on the other hand, you aren't tied to whatever version the system wants to give you, you can keep running the current version until you're ready to upgrade.

I'm looking forward to Shopify's part 2 blog post to see what they do exactly. :)

> sshd (or any other more common program; I'm just using sshd as an example to understand the rest of the system) won't magically know to look at /nix/store/efgh5678-my-sshd-config-2.0/etc/ssh_config, right?

Conceptually, my-sshd-config includes a script (or systemd unit, or whatever) that has a reference to a particular version of sshd and also has your config, and so it would run "/nix/store/aaaa1111-openssh-9.0/bin/sshd -f /nix/store/efgh5678-my-sshd-config-2.0/etc/sshd_config". The openssh package doesn't know about you, and you can't change it, but you know about it. (In other words, the inputs that resulted in the hash efgh5678 include "aaaa1111-openssh-9.0".)

It looks like the actual way you do this in NixOS is that the sshd package provides a function in the Nix language which takes some config as input and spits out a package as output. So your systemwide config file loads the sshd package and calls a function, which returns a systemd unit with the right filenames. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/s...

[mehrdadn]

Wow interesting, okay. I think I'll probably have to give it a shot at some point to try it out. Hopefully it'll live up to the expectations :-) thanks a ton for all the explanations!

[bb010g]

Have you looked into https://github.com/rycee/home-manager ?