[jchw]

I think you may be overthinking things. In brief;

Nix is two things: a functional programming language called Nix, and a package manager written in C++ that compiles and installs packages written with Nix expressions. Nix the language itself, and it's implementation.

Nixpkgs is a repository of Nix packages. It also contains NixOS, which is also just a bunch of Nix.

NixOS is a Linux-based operating system built on Nix. It is configured using NixOS modules, a configuration system of sorts built on top of the Nix programming language. A NixOS system is literally a single big Nix package built from NixOS modules that depends on a bunch of other packages.

If this still seems too complicated it would probably be better to just dive in and mess with it so you can physically see what's going on.

> current problem is taking a homelab server into “production” and all I really want is a way to write down a state I want the services on it to be in, have that in a git repo and be able to regenerate this server after loosing it (in a first step without any data backup). I’m still, after 2 months (wall clock, call it maybe 3 days actually getting into this project), not sure if that’s a thing Nix tries to cover! (I think it does, but..)

Yes that does sound like something NixOS can cover. Obviously, you will still need backups for your mutable data (databases, files, etc.), but if you are able to encapsulate the entire setup of a server into a NixOS config you can indeed throw a machine out the window and have a new one set up and booting the exact same services in minutes.

This is pretty much the original thing that drew me into NixOS, especially because at the time I was highly interested in qemu GPU passthrough setups and I really liked how with Nix all of the complicated setup bits could be expressed in a single file, with about 20 lines of Nix, all accounted for and version controlled. A lot better than just hoping you remember to keep things documented.

I won't lie, learning NixOS and using it for a real machine setup was very painful for me at first. I struggled with many things over time. Still, I really think because the scope of things these projects do is huge it's really easy to overcomplicate things. All of the explosive scope of Nix comes from relatively simple primitives applied in large systems.

I really don't want to encourage people to suffer with Nix if they feel like it's just too convoluted and hard to work with. I am not trying to convey that Nix is simple or easy on the whole. I was only ever trying to explain why I felt the issues raised by this post were confusing to me, not trying to tell people that Nix is generally easy. I think a lot of replies genuinely continue to misunderstand this, but the point of saying that it's easy to do that in Nixpkgs is not to dunk on people who struggle with Nix, it's to say that it's not clear why Nixpacks would struggle with these particular issues. You can literally just split the /nix/store paths into separate OCI layers. There is no obvious reason why this strategy can't be used, and the blog post doesn't go into the background behind it. I bet there actually is background, it's just left unsaid.

[hnaccount_rng]

I think I got the first part, including and up to

> If this still seems too complicated it would probably be better to just dive in and mess with it so you can physically see what's going on.

And it really seems like NixOS being _the_ correct :tm: solution for what I want from an home server. The problem is that Nix doesn't quite cover (or at least didn't when I first deeply looked) my needs (it lacked a useful Clickhouse package). And I don't even want to say "Nix should be simpler". I don't even think it _can_, it's fundamentally a complicated and complex problem to deal with!

My problem is solely with the Nix community. The amount of jargon-loving and argumentative "you shouldn't attempt what you are attempting in the first place" or "you are holding it wrong" gaslighting is really something :D

I think the point where you fail to grasp the problem is: Nobody is claiming that Nix can't do a thing. My complaint is: There is no way to find _the_ way to do the thing. And there seem to be multiple _incompatible_ ways to achieve any given state. Without any way to distinguish those (let's call them) camps for a new person. In this way it's everything Unix but dialled to eleven! You don't need to argue (for me) that anything _can_ be done. I'm getting that from "Turing complete". My problem is only the practice: How do I get there?

edit: I think my immediate problem is: How do I get the series of "commands" that a nix derivation will invoke? Kind of like "what's the receipt, that is being executed?" That's missing for a sensible repl-loop for me. I'm sure I'll be getting there. The capability that Nix offers is extremely appealing. But at the same time the documentation and community are unhelpful to the extreme

[jchw]

> My problem is solely with the Nix community. The amount of jargon-loving and argumentative "you shouldn't attempt what you are attempting in the first place" or "you are holding it wrong" gaslighting is really something :D

> I think the point where you fail to grasp the problem is: Nobody is claiming that Nix can't do a thing. My complaint is: There is no way to find _the_ way to do the thing. And there seem to be multiple _incompatible_ ways to achieve any given state. Without any way to distinguish those (let's call them) camps for a new person. In this way it's everything Unix but dialled to eleven! You don't need to argue (for me) that anything _can_ be done. I'm getting that from "Turing complete". My problem is only the practice: How do I get there?

I honestly have relatively little experience with the majority of the Nix community, so I can't speak too much to it. I did not personally experience too many problems.

The frustrating thing to me about this thread is that it definitely wasn't in this vein. I know I'm harping on this a lot, but it matters.

What the blog post says is "we can't split the image into multiple layers with Nix". And again, they're doing their own OCI building, so this actually has little to do with Nixpkgs and is much more conceptual in nature. If you build a Nix closure, what you wind up with is a number of store paths. For example, for bash, you might have the following store paths:

    /nix/store/za53jjhjl1xajv3y1zpjvr9mh4w0c1ay-xgcc-14.2.1.20250322-libgcc
    /nix/store/yypqcvqhnv8y4zpicgxdigp3giq81gzb-libunistring-1.3
    /nix/store/6nkqdqzpa75514lhglgnjs5k4dklw4sb-libidn2-2.3.8
    /nix/store/cg9s562sa33k78m63njfn1rw47dp9z0i-glibc-2.40-66
    /nix/store/xy4jjgw87sbgwylm5kn047d9gkbhsr9x-bash-5.2p37

So how do you split these into layers? Well, however you want. You can place them into individual layers, or you can group them based on dependencies to make more "logical" groupings and get less layers, or basically any number of things. There's plenty of metadata on what depends on what, so it's really up to you to decide how to do that.

I was not talking to end users when I said this, I was talking to developers building on Nix that almost definitely knew this. So it's unclear based on what they said why they couldn't.

> edit: I think my immediate problem is: How do I get the series of "commands" that a nix derivation will invoke? Kind of like "what's the receipt, that is being executed?" That's missing for a sensible repl-loop for me. I'm sure I'll be getting there. The capability that Nix offers is extremely appealing. But at the same time the documentation and community are unhelpful to the extreme

I see.

Well, for starters, you are trying to start at the very low-level bottom of the stack. I understand this because it is the approach I prefer to take when learning new technology, but I'm telling you right now this is a hard road to go down with Nix, I've been down it. The documentation is going to continue to feel unhelpful because you will find plenty of documentation that is happy to take you down the dark paths of how things actually work, but that doesn't make them any easier to navigate. It's much easier to start by asking "how do I accomplish this end-goal" than to ask "what actually is a derivation".

With that out of the way... to understand what's going on with builds, you really genuinely do need to know what a derivation is, because it matters. Almost nobody calls the `derivation` function directly in Nix, instead using Nixpkgs mkDerivation tool at the bare minimum, but at the very bottom, there is only one function that matters:

https://nix.dev/manual/nix/2.22/language/derivations

At the bottom, in order to actually build something, a derivation has to be evaluated and realized. Internally, instantiating a derivation will cause it to be built. It is built by taking the parameters passed to derivation and serializing them to an internal format, then "instantiating" it, causing it to be built inside of the sandbox. The internal derivation format looks like this:

    Derive([("debug","/nix/store/xsxanvyy93g3k5dnh0j2wisrwvq012b2-bash-5.2p37-debug","",""),("dev","/nix/store/lnzar8zawj43wkqlsdqhywda9hjymrgf-bash-5.2p37-dev","",""),...

Basically a lot of arrays and tuples and whatnot. Not very easy to read. `nix derivation show` can give us better human-readable output:

    {
      "/nix/store/rfkzz952hz2d58d90mscxvk87v5wa5bz-bash-5.2p37.drv": {
        "args": [
          "-e",
          "/nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh",
          "/nix/store/shkw4qm9qcw5sc5n1k5jznc83ny02r39-default-builder.sh"
        ],
        "builder": "/nix/store/razasrvdg7ckplfmvdxv4ia3wbayr94s-bootstrap-tools/bin/bash",
        "env": {
          "NIX_CFLAGS_COMPILE": ...

So what does `nix build nixpkgs#bash` do? Well, aside from the sandbox setup, on my particular machine, it would run... /nix/store/razasrvdg7ckplfmvdxv4ia3wbayr94s-bootstrap-tools/bin/bash -e /nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh /nix/store/shkw4qm9qcw5sc5n1k5jznc83ny02r39-default-builder.sh. So, basically, a bash script. What does that bash script look like? Here's the entire contents of default-builder:

    genericBuild

This is in the heart of the Nixpkgs "stdenv" machinery, and now we're outside of what Nix itself does and into what Nixpkgs does. Literally all Nix does is handle the book-keeping: sandboxing, execing the builder, handling the Nix store; for the actual build, it is just execing the builder, and in most cases, it will wind up being handled by a bash script. You could have whatever builder you want here, of course, and it needn't be bash or bash-based. (Note that we're calling bash to build bash - funny choice for me to pick, but you can see how it gets the copy of bash to build bash with from a special derivation called bootstrap-tools. Gotta break the cycle somehow, and Nix doesn't have as impressive of a bootstrap seed as Guix.)

So, for this build, it just calls genericBuild. What does that do? Well, you can keep chasing down the scripting here, but ultimately it calls all of the build phases, e.g. unpackPhase, patchPhase, configurePhase, buildPhase, checkPhase, etc. which are all just bash scripts inside of environment variables. At the end of the day, it does roughly what you would do to build bash, plus some patches. The nixpkgs stdenv automation helps handle many special Nix things, but it's not actually horribly complicated.

By the way, when you see this sort of thing in a Nixpkgs derivation:

    patchPhase = '''
        ...
    ''';

It is in fact just setting up environment variables. You can see them in `nix derivation show`, even. They get called just by substituting them directly into a command, not any different than:

    test="echo Hello world" bash -c "$test"

Basically, all of the attributes in the mkDerivation call become environment variables, including src, and all of the magic that gives them meaning is just in the stdenv default builder. (This is roughly true, anyway. I think there is some move to make it "stricter" but the default way it works is still like this roughly.)

So how can you debug what actually happens when building a derivation? Unfortunately it's not terribly easy. You can view the logs with `nix log`, but it will only show one if it actually built it, not if it substituted it from Hydra cache for example. Also, the default behavior of the builder doesn't output that much diagnostics, though you can add NIX_DEBUG = 7 in a derivation and it will output tons of information about what is being executed.

The way derivations depend on other derivations is something like this:

- During build-time, when you try to coerce the result of the derivation function to a string, what gets returned is the realized store path. So if I try to do `${pkg.bash}/bin/bash` inside of Nix, it might return `/nix/store/xy4jjgw87sbgwylm5kn047d9gkbhsr9x-bash-5.2p37/bin/bash`. In this way, it is natural to literally just depend on something implicitly by realizing it as needed.

- As far as I know, for runtime dependencies, it is detected by literally scanning the $out directory after a derivation is built and looking for the Nix store paths of other derivations. Merely realizing a dependency during build-time won't make it a runtime dependency.

It's more complicated than this, but that's the broad strokes. So when you build bash, it will implicitly depend on all kinds of build tools like GCC and make, but the resulting derivation will only depend on things like libgcc that are actually referenced in the final derivation.

Honestly, it is a bit of a black box. It is not easy to navigate. Trying to learn Nix from the ground up first will be very hard. I wouldn't ever tell someone not to try, but I will definitely say it won't be easy.