[whatwhaaaaat]

Just for a giggles, what projects do you work on that require such variation in tooling where something like this becomes worthwhile?

I always see these type of arguments for why nix is so great but it’s never been a pain point for me in 10+ languages and 20 years of development experience. I see your example of bash scripts but this can’t be all for writing scripts.

[tstrimple]

Not the OP, but I work in consulting. When I was still hands on keyboard, this would have been very helpful for the clients who don’t provide their own hardware or environment for us to use. I also do work for extremely large organizations who have literally dozens of different stacks accumulated over the decades.

In addition, I play with all sorts of open source tools and they often come with their own tool chains and expectations. Python version management in particular benefits a lot from this level of isolation. Instead of figuring out the different version management tools for each stack I use a higher order environment management tool in Nix.

Some others are solving these issues with containers, and that’s a part of the nix strategy as well.

[kfajdsl]

I've previously used Nix to manage C/C++ projects and ended up with a really nice flow, so I really want to use Nix for Python, since I've had so many issues with conda. However, every time I've tried, I've ran into enough issues trying to get a lot of ML packages I use to work (dealing with transitive dependencies on esoteric packages, mostly) that I couldn't justify continuing rather than just hacking my way to getting the conda environment working with random pip packages, pinned versions, etc.

[otabdeveloper4]

Conda is a bad idea, eventually conda will crap the bed and start hanging indefinitely. The cool kids don't use conda anymore.

[__MatrixMan__]

I've been considering an AI project for consuming a conda build recipe and digging into the codebase to extract extra info about the project and make it into a nix flake--which would be a bit more stable. I figure you could test for equivalence in a few ways and feed the outputs of that test back into the model. Hopefully there's enough context in the conda recipe, the project codebase, and whatever errors pop out to get some of them converted with minimal handholding.

Because regardless of what the cool kids are doing, important work is being done in conda, and usually by people whose expertise isn't software packaging.

[whatwhaaaaat]

Yeah I get the idea but I’m asking op for concrete examples. Python has its own environment management options that work well. I’ve read on this site over and over what it can do - I’m wondering if anyone has hard examples of tooling they switch about enough to make it worthwhile.

[otabdeveloper4]

If Python environment management works well for you, then stick with that. You'd be in the minority, though.

Nix is a pain in the ass, people mostly use it because it's the only reliable option.

[lmm]

Scripts are the main place where it matters tbh - most language ecosystems have their own way of doing this stuff, if you can stay within the language you're fine. But if you (or your client) have a culture where people throw in awk/grep/sed then there's just no real alternative. Or if it's a polyglot project where you have three different languages (including shell) then you may not be able to use a single language package manager.

[__MatrixMan__]

> if you can stay within the language you're fine

Agreed, but people have a tendency to do otherwise. At the very least you still have to install the right version of the language. And there are probably an few other tools, linters and such... Next thing you know you've got quite a pile that's not covered by your language's package manager.

[lmm]

I find different languages/ecosystems have different cultures around this. In Java/Maven land it's fairly common to have a self-contained project where all the helpers like linters etc. are set up in Maven so all you need is a vaguely recent JVM and vaguely recent Maven. But there are other ecosystems where people like to throw a bunch of shell scripts etc. in.

[__MatrixMan__]

When Python comes up as an example of a problematic packaging ecosystem, Java often comes up as an example of it being done right. I think the key is the cultural difference you're pointing to. JVM folk are not tempted to stray from the JVM. Python folk think of Python as a convenient harness for that cumbersome bit of FORTRAN that they can't live without.

I only worked in a Java shop once, but I remember that they looked at me like I was an alien when I proposed that we involve a subprocess written in a different language.

At the time I thought they were insane for writing everything themselves but I've since seen how gnarly packaging can get and now think that they're... less insane.

[__MatrixMan__]

The most recent offenders were nodejs and kustomize used as part of a test flow orchestrated by a Makefile, run both locally and in CircleCI.

People will just install the latest version and start hacking away, and now you've got all this code that depends on that version. Backwards compatibility ain't perfect, so maybe several years later the original author doesn't work here anymore, so tests are breaking in subtle ways when you install what's the now latest version and there's nobody to ask what the "right" version is.

But since we're a culture that uses these tools (though I wish we weren't), this story has played out several times so different projects need different versions--you can't just discover the right one once and leave it installed in your system, you have to install the right one for your project and change it when you switch to a new project.

For the most part these are go projects, so even though there is language-specific dependency locking via go.mod and such, dependencies which aren't go libraries but which are nonetheless needed to work with the project (e.g. make) are left as an exercise to the reader. Make is pretty well behaved, I haven't had to do much version antics with it, but I wouldn't say that's the norm.

When I find one of these repos I put my archaeologist hat on and write a flake.nix to provide whatever the dependency is, and then I walk the version backwards until it starts working. That way next time I'm in that project I don't have to go through that exercise again.

To make matters worse, people often try to help by adding entries to the makefiles which download the correct version for that project, but some people have the newer arm chips and others are still on x86 so confusion of a new kind ensues. Of course it's easy to fix these scripts to detect the local architecture, but that's a whole extra step.

And then maybe you're trying to make this stuff work in CircleCI or somesuch, and you don't want the workflow to just be reaching out via https and blindly running whatever comes across the wire because who knows if it'll be the same thing tomorrow, so you add hash checks, but once you've got the hash checks and the architecture checks and you're checking the right hash for your architecture... we'll you've basically got a poor man's flake.lock at that point, might as well use the same nix config in both places rather than use homebrew or apt or whatever locally and then figure out how to do the same thing via circleci orbs in yaml and god forbid you have to do it in prod too so now there's a Dockerfile... Having a single source of truth for dependencies and using it everywhere is super handy.

That's work. Another example is in my personal projects. I use helix, so there's a .helix/languages.toml in my repo which defines the language servers that I'd like it to use for that project. But merely pointing helix at the language server isn't enough, it also has to be on the PATH. My older projects are using mypy, and my newer ones are using pyright (python type checkers).

Sure I guess I could just install both at the system level everywhere I go, but when I clone the project on a new machine I want to have everything work right away--I don't want to start coding and then wonder why my editor sucks and then go discover and install the right LSP and then resume coding. I'd end up with a smattering of different versions installed across all of my devices, even for the same project. If I find a bug which happens on this machine but not that one, I'd have a much harder time knowing where to start re: debugging it.

Finally there's this idea that maybe you don't even have to clone the repo, you can just reference the app you want to run from anywhere. I invoke some of my tools like:

    nix run git+ssh://[e-mail]/myorg/myrepo -- mycli --best-arg

Nix knows how to set up the environment for running the app (the one I have in mind is written in python, so a certain version of poetry is involved etc...) so I really don't need the caller to think about that environment at all. I like this because it decouples the orchestrator from the executor. So if I can manage to get something working locally with one of these commands, then I can go put the command someplace weird like in an Airflow DAG (kubernetes pod operator, NixOS image) and I have a pretty strong assurance that it'll work the same in the remote environment as it does locally.

From the perspective of a nix user, these problems are all the same problem and they're everywhere and nix is the only thing that solves all of them at once. My feeling is that from the outside, they look like separate problems, and it's not clear just how many of them are solved by nix--so the juice doesn't appear to be worth the squeeze.