[cloudripper]

I gave LFS a go earlier this year. I learned a lot through the process - but I definitely went outside the guardrails. I use NixOS as my daily driver and found myself curious of whether I could complete LFS using a "Nix" approach. I was only a basic Nix user at that time and that choice made a difficult process much more difficult. However, the declarative nature of Nix meant that I had clear notes of every step of my process - and if something didn't work, I could backtrack and troubleshoot to find the root of the cause. The end result is here [0].

My understanding of Linux, of bootstrapping, cross-compilation, and Nix has grown tremendously as a result of the time I took on this project - and I still go back and reference the work from time to time. When I get some time to revisit the Nix-based LFS project, there are quite a few things I would like to clean-up, including setting kernel configs and handling post-build permissions.

Nix-complexities aside, I highly recommend LFS if you like to understand how things work and don't mind a little suffering along the way.

[0]: https://github.com/cloudripper/NixLFS

[hi-v-rocknroll]

While Nix/NixOS solved RPM dependency hell and side-by-side multiple concurrent version installation issues, it created its own problems.

The problem with Nix/NixOS is everything is too isolated to be configurable or usable, and so almost everything is broken because it doesn't work like any other OS. Chasing down all of the oddities and gotcha problems becomes a full-time IT job that slows everything else down, and then with a fragile special snowflake of patches and workarounds, it becomes very expensive and slow to do anything or to count on it being reliable. Furthermore, the syntax and nature of the packaging DSL is also too clever special snowflake that could've made do with declarative data or imperative procedural with something widespread like Python, Bourne shell, or JS to reduce the friction of customization and contribution.

Like Qubes, an over-optimization for particular goal(s) ends up becoming Achille's footguns because of a lack of compatibility and usability with everything else that came before.

[ertian]

You're basically just saying it's unfamiliar, and not universal yet.

In my experience, using & configuring NixOS is much simpler and easier than most distros...until it's not, at which point, yes, the difficulty curve is very steep.

Even in those cases, though, it's got the nice feature that once it's solved, it's solved. All the config is in one place, and you can apply the same config across all your hosts. You don't get those situations where you get something working after tweaking configs, env vars and systems jobs, and can't remember how to reproduce it.

And though some of the conventions around NixOS are pretty arcane and convoluted (though the language itself is pretty simple), it has the advantage that you only need to learn it once and you can use it all over, instead of switching between bash, python, yaml, toml, systemd units and even more obscure formats (nginx, apache, postfix, etc).

IMHO, the tradeoff is completely worth it.

[VTimofeenko]

Chasing down oddities on predictably isolated environments is much easier though. And since the environment is declarative, one can actually roll back to the previous working state without breaking other things. For that alone the learning curve of the DSL is worth it IMHO, but I am not saying that the language and ecosystem are perfect just the way they are.

[therein]

If the DSL was easier to work with, more familiar; and if there was some good IDE support with autocomplete and documentation, I think it would be amazing.

[tomberek]

This is excellent. Have you considered making a presentation or a write-up of the experience?

[cloudripper]

Thanks for the input. A couple of folks suggested that recently as well. Once I can clear up some bandwidth, I do intend to follow through on that. It gave me a huge appreciation for Nix as a build system, and I would love to share that if it were helpful to others.

[ayakang31415]

The first thing that pops out on Google search of NixOS is "Declarative builds and deployments." What exactly is NixOS different from other distros, such as Ubuntu?

[hollerith]

NixOS separates packages. If the package foo contains a file /usr/bin/foo, NixOS installs it in /nix/store/67c25d7ad7b2b64c67c25d7ad7b2b64c-foo/usr/bin/foo. In order to make this separation work, Nix must sometimes rewrite binaries so that all references in the binary to /usr/bin/foo becomes references to /nix/store/67c25d7ad7b2b64c67c25d7ad7b2b64c-foo/usr/bin/foo.

The advantage of this approach is that it gives more control to the distro maintainers and the admins of the computer, taking that control away from the "upstream" maintainers of the software being packaged. For example the software being packaged cannot just call the library bar because bar is not at /usr/lib/bar.so like it is in most Linux distros -- it is at /nix/store/17813e8b97b84e0317813e8b97b84e03-bar/usr/lib/bar.so, but of course the software does not know that unless the person creating the Nix package (the packager) arranges for the software to know it (again sometimes by doing a search-and-replace on binaries).

If the upstream maintainer of foo thinks foo should link to version 6 of library bar, but you think it should link to version 5, NixOS makes it easier for you to arrange for foo to link to version 5 than most distros do (even if version 6 of bar is needed by other packages you have installed which you need to use at the same times as your using foo).

Note that if this separation of packages imposed by NixOS has any beneficial security properties, it is merely security through obscurity because there is nothing preventing a binary from covertly searching through the directory listing of /nix/store/ for the name of the library it wants to call. Nevertheless it turns out the be useful to seize some control away from upstream in this way even if technically upstream could seize the control back if it were willing to complicate the software to do so.

People, including the creator of Nix and NixOS (Dolstra), will tell you that NixOS's main advantage is "declarativeness" (which in the past Dolstra called "purity") or the fact that the compilation / building process is deterministic. I believe both positions (NixOS's advantage is declarativeness and the advantage is deterministic builds) are wrong. Specifically, I believe that although deterministic builds are useful, the separation of packages I described is much more useful to most users and prospective users of NixOS.

Another way to summarize it is that NixOS package maintainers routinely modify the software they are packaging to use less "ambient authority".

[_ank_it]

Extremely noob here, I was trying nixos and got real confused about how to install python packages as pip was not allowed at system level.

[j0057]

If other distros allow pip-installing into the system, that could be considered a bug or at least an anti-feature, because it's almost always a bad idea: it can clash with distro-managed Python packages, it will break on Python upgrades, and sooner or later you will run into version conflicts (a.k.a. DLL Hell). Recent versions of pip refuse to install into the system by default, for all of these reasons.

It's better to instead pip-install Python packages into virtual environments, recent Pythons havr `venv` built in for this purpose. For user-scoped or system-scoped utilities, `pipx` can manage dedicated virtual environments and symlink them into the search path.

[imiric]

Nix purists would say that you should use flakes to declare all the dependencies for each project, and reference all Python dependencies as Nix packages there. Nix effectively tries to replace every package manager in existence, so all Python, Ruby, Emacs, etc. dependency trees are duplicated in Nix.

I think this is insane. Not only will many packages be missing from Nix, you will also have to wait for the upstream changes to actually propagate to Nix repositories. This all assumes, of course, that there are no packaging issues or incompatibilities in this repackaging.

This is one of the ways that Nix sometimes just gets in your way. I've been using Nix(OS) for several years now, and this still bothers me.

Instead of doing this, For Python specifically I would suggest installing pyenv, which Nix packages. Then enter a nix-shell with a derivation thingie[1,2], and install Python as usual with pyenv. Then you can use any Python version, and with pyenv-virtualenv (which Nix _doesn't_ package...), you can use venvs as you're used to. Sure, you don't get the benefits of the declarative approach and isolation as with "the Nix way", and you may run into other issues, but at least it's a workflow well known to Python devs. Hope it helps!

[1]: https://gist.github.com/imiric/3422258c4df9dacb4128ff94d31e9...

[2]: It took me way longer than I would like to admit to figure this out... This shouldn't be so difficult!

[therein]

Ironically, that's when you create an Arch container in LXC.

[hollerith]

I would try installing pacman in the same filesystem NixOS is in.

[imiric]

Meh, maybe. Containers make everything more difficult for local development, though. That would be my last resort in this case.

[nextos]

The right way is to compose the base Python package with the libraries you want. For example, this gives you an ephemeral environment with the latest python3 plus NumPy:

  nix-shell -p 'python3.withPackages(ps: [ ps.numpy ])'

In case of Python, you can also go for a simpler option that avoids composing Python with its packages, but that gives worse isolation:

  nix-shell -p python3 python3Packages.numpy

If other packages were present in that ephemeral environment, aside from python3, they could see NumPy in a global directory. That's why the first option is preferable, as it offers better isolation. In Nix, some languages only let you use libraries with something like the first option. See the Nix user wiki for further information: https://nixos.wiki/wiki/Python.

[zenoprax]

I'm new as well - it's good to remember that NixOS, nix-shell, and the programming language of Nix are all separate. You can start with your current distro to learn nix-shell first.

I still have no idea how it all works but it seemed prudent for me to at least try.

[rcxdude]

The correct answer, whatever distro, is to either use the system package manager or use python's venvs. Mixing multiple package managers in a given area of the filesystem is a recipe for breakage.

[MuffinFlavored]

Why did you need to chmod every ~2,600 files? man pages, docs, includes. Why not chmod -R or something?

https://github.com/cloudripper/NixLFS/blob/64e333f60cca4bec5...

[cloudripper]

Fair questions. If the question is "why did I need to chmod those files", the answer is that permissions modifications in the mkderivation build environment are not propagated to the build output. The quick and dirty solution I came up with was to make a chmod wrapper that logged the calls so that I could apply them after the entire build completed.

For why I did not chmod the directories - that is something I should probably do when I get time again (cleaning up permissions handling more broadly). In my case at the time, I had the logs from the wrappers and did a quick filter before directly adding them to the script as a last hurdle to having a bootable machine after a month-long grind.

The kernel config handling needs a good amount of attention too. I have a cleaner approach to that in mind that I haven't had the opportunity to work on.

It will likely be December before I am able to invest much more time into it, but definitely open to any input generally.

[MuffinFlavored]

> The kernel config handling needs a good amount of attention too.

https://github.com/cloudripper/NixLFS/blob/64e333f60cca4bec5...

Could you speak a bit about why you did it this way instead of maintaining a complete copy of the kernel .config you wanted?

[cloudripper]

There's a couple reasons for that.

First, for my initial attempt at the project I wanted to make as close of an adaptation from LFS guidance to Nix that I could. While I did sprinkle in some custom kernel configurations (from BLFS guidance), I otherwise tried to make a direct adaption from the book. The LFS guidance for the kernel build uses `make menuconfig`. That requires user interaction which, as far as I am aware, is not feasible in a Nix derivation (and really wouldn't make sense to achieve reproducibility as far as I understand the term). So I used `make defconfig` within the derivation and attempted to sed the kernel .config to achieve a matching config to that of LFS guidance (coupled with a few sprinkles of custom configs). In retrospect, I don't like how that block currently flows and would like to make it a bit cleaner and easier to declare configuration preferences.

Second, my approach with the entire project was to limit external inputs to the derivations with the goal of trying to be as reproducible as I could. If I recall correctly, the only external inputs used in the whole project were the sources and patches provided by the LFS project. That said, I hadn't considered that it might be a lot simpler to set it up in a "Bring Your Own Config" kind of way. I will stew on that more and would definitely be open to more thoughts on either and/or another approach.

[MuffinFlavored]

Well, for as critical as I might sound, you did absolutely awesome and the project literally inspires awe in me. I look forward to trying it. It is really, really cool and well done so thank you again.

[danlitt]

oh, fun, I was planning to do the same thing!

[ocean_moist]

> This results in a significant storage footprint, often exceeding 12GB.

12GB being considered significant makes me feel good, a return to simplicity. The other day I couldn't provision a VM with less than 512GB of storage...

I can't even play my favorite games without giving up >150GBs...