[scandinavian]

Now that we have another Nix post, maybe someone can enlighten me about something I've been wondering about.

I'm one of the maintainers of a popular django application. Someone made a nix package of the project, but we've now twice gotten invalid bug reports from people using the package because the package depends on "django_4" and whenever someone updates that nix package, the package for our project breaks.

Of course we, like all other python projects, don't support using other dependency versions then the ones in the requirements.txt file. So when someone just uses a different minor version of django, stuff breaks. What's the disconnect here? Why does all nix packages that use django_4 need to use the same version, that seems super prone to breaking all kinds of stuff. Same for the other 35+ dependencies that run arbitrary versions instead of the ones defined in the requirements.txt file.

[matklad]

I am not an expert, but here’s my attempt at a useful comment.

On the highest level, `nix` is an alternative build system. So, if someone packages your app with `nix`, there’s now extra work to keep that working, and it’s on the packager to keep it working. If they packaged your app such that it’s using different dependencies than those required, that’s a bug in the package. As a maintainer, you can help here by making it clearer what versions are accepted, and by making it easier to run the tests for a package.

If we open a black box, there are two things in play here: Nix-the-build-system and nixpkgs package collection.

The build system is very open ended and can specify all dependencies precisely, but it’s on the user to define what that means exactly.

nixpkgs is a coherent collection of nix packages, a bit like a Linux distro. In particular, it _generally_ has one version of each package, and there’s some testing to make sure that all the packages work together.

Now, to package a Python app with Nix you can either pull dependencies from nixpkgs, in which case the situation would be similar to, eg, packaging for Debian.

Or you could create a hermetic environment, where an app gets an isolated copy of dependencies, specific just to the single app, a situation similar to using virtual env.

It sounds like what happened here is that your app got packaged in the fist way, but actually it can work only in the second way. I assume you do specify specific compatible version of Django somewhere, and if a package (be it .deb, .rpm, or .nix) doesn’t respect that, that’s a bug in the package.

Hope this helps!

[SuperSandro2000]

> packaging for Debian.

Not really, nix is way more flexible and more up to date and nix also often runs tests and different pythons cannot interfere with each other that easily. On a high level things are similar but the details are wastly different.

> Or you could create a hermetic environment, where an app gets an isolated copy of dependencies, specific just to the single app, a situation similar to using virtual env.

That could also be done with nix but is often not because upstream pin quality is often lacking.

[hamandcheese]

It seems like Nixpkgs aims to minimize the number of package versions in use at one time. Not just nix, most package managers do, it seems (i.e. you wouldn't expect to find different minor versions of Nginx in Debian, would you?)

So by that same logic, there is only one version of Django 4.

It is definitely possible with Nix to use the precise versions of what's in your requirements.txt, but I'm not sure if the Nixpkgs maintainers would allow all that extra duplication upstream.

[traxys]

I packaged some python applications in nixpkgs, and it seems the consensus is to try and relax the dependency so that the globally packaged version is used, but if it fails the you can override the version yourself. Though this is not done through the requirements.txt because that file does not have enough information (no integrity hash for example).

[scandinavian]

I get what you are saying, but nothing you said works in practice for python packages, so not sure that I actually learned anything.

Is it fair to summize that python applications with python dependencies do not really work well as nix packages and shouldn't be used?

[hamandcheese]

> but nothing you said works in practice for python packages

How do transitive dependencies in the Python ecosystem work, then? I assume Django works with multiple versions of python and bcrypt. I assume pandas works with multiple versions of scipy. Is there no semantic versioning? If everything requires an exact version, how do you prevent everything from grinding to a halt?

> Is it fair to summize that python applications with python dependencies do not really work well as nix packages and shouldn't be used?

Let's not conflate Nix and Nixpkgs. Nixpkgs has its reasons for minimizing redundant packages, however it is certainly possible to package your app with Nix and use the exact specified dependencies.

[scandinavian]

> How do transitive dependencies in the Python ecosystem work, then?

Not very well.

> how do you prevent everything from grinding to a halt?

I don't have a good answer for you.

> Is there no semantic versioning?

You can read django release process here [1], not sure how it's relevant. I'm not the maintainer of django, but of a project using django. Would it be better if all software was perfect, had no bugs and used perfect semantic versioning? Yes, I would say so. Is that a requirement for using nixpkgs?

> Nixpkgs has its reasons for minimizing redundant packages, however it is certainly possible to package your app with Nix and use the exact specified dependencies.

I'm not packaging it, someone else is, it breaks and they come to the project to raise invalid bug reports.

[1] https://docs.djangoproject.com/en/dev/internals/release-proc...

[hamandcheese]

> not sure how it's relevant.

Well you said earlier that nothing I said works in practice for python packages. My only point is that it must work at some level in the python ecosystem, else the ecosystem would collapse.

Anyways, it sounds like you're unhappy that someone did a bad job packaging your application. That sucks. Elsewhere in this thread someone mentioned that there isn't a strict single version policy in nixpkgs, so this can probably be easily fixed. I'd suggest filing a bug in Nixpkgs.

[SuperSandro2000]

> Elsewhere in this thread someone mentioned that there isn't a strict single version policy in nixpkgs, so this can probably be easily fixed. I'd suggest filing a bug in Nixpkgs.

There isn't one but we are not collecting multiple package versions for no reason and since python itself cannot well handle multiple versions of packages they are only allowed outside of pythonPackages where all end user applications should live.

[eulenteufel]

In this case I think it is important to distinguish nix (the package manager) and nixpkgs (the popular package repository / distribution used with nix).

Packaging python applications with nix is doable, but you have to specify the exact versions of your dependencies and for that you can't easily use nixpkgs.

Nixpkgs tries to keep a minimum number of packages (like Arch or Debian as well), so each of the dependencies will typically only occur with one minor version for each release of nixpkgs.

We could still use the nixpkgs to build our application but we have to override each of our dependencies to the right version, but that approach can get quiet tedious for a large number of dependencies.

Fortunately there are tools to automatically generate your dependencies from a requirements.txt such as mach-nix or pip2nix.

[eptcyka]

What's the point of a minor version change if it's breaking? Does Django not have a versioning policy that enforces non-breaking changes between minor versions?

[scandinavian]

That's my fault for writing minor, as that's what it would be in semver. I should have written feature release.

You can read the release process here.

https://docs.djangoproject.com/en/dev/internals/release-proc...

[KirillPanov]

> python dependencies do not really work well

Yes, that is exactly correct.

[ruuda]

No, that is not a fair summary; Nix is the nicest way to manage Python packages that I have found thus far.

[ParetoOptimal]

I'm assuming you try to keep all dependencies on the nixpkgs version?

[Cu3PO42]

That's not really necessary. Pulling in arbitrary versions of packages from PyPi is fairly easy, if a bit verbose.

[ruuda]

For packages that I need to pin to specific versions, that's also easy to do with Nix.

[SuperSandro2000]

> Is it fair to summize that python applications with python dependencies do not really work well as nix packages and shouldn't be used?

No, applications that are properly maintained work as they should and this can be ensured with tests and e2e tests.

[lolinder]

This is such a condescending attitude. What you mean is applications that are maintained the way that you and the Nix developers think an application should be maintained.

It's incredibly naive for a package manager as ambitious as Nix to assume semver. I'm a big fan of semver myself, but the vast majority of software projects follow it imperfectly or not at all, and for good reason—it's nearly impossible to follow it perfectly, because even bugs are part of your API. Every project I've worked on has eventually had something break on a version upgrade because we were depending on something that was later decided to be a bug (but at the time was just how it worked).

Elm can mostly get away with enforcing semver because they designed it that way at the language level, but Nix wants to manage dependencies written in all languages and ecosystems, which have dramatically different versioning practices.

[hamandcheese]

I think the thrust of the parent comment was more that the test coverage of this package isn't good, not that semver must be followed.

[lolinder]

Ah, fair enough, I misunderstood. I thought the tests they were recommending were tests to ensure backwards-compatibility between version bumps, I didn't realize they were talking about the downstream pacakge's tests.

I still disagree with the insinuation that it's everyone else who's screwing up and if we all did things the way Nix wants us to then Nix would actually work just fine. That's just another way of saying Nix doesn't work in the real world.

[SuperSandro2000]

Nixpkgs does not assume Denver that's why we run if possible the package's tests, our own tests and build dependent packages to make sure the most obvious breakages are noticed before things are even merged.

[lolinder]

Ah, I thought you were saying that if we all just used e2e tests to ensure we didn't make breaking changes in minor versions, we'd be fine. I didn't realize you were talking about the downstream package's tests.

I do still take issue with your insinuation that it's the package maintainers' poor practices that are at fault here. The real world is a messy, complex place and "best practices" don't translate well from situation to situation.

OP didn't ask for their package to be included in Nix. Presumably OP's system works for them and for their use case, but whoever created the Nix package made assumptions that turned out to be flawed. It's not fair of you to say that those bad assumptions are OP's fault because their package isn't "properly maintained" and doesn't "work as it should".

Someone (you?) made a bad assumption. Don't cast blame for that on someone who only knows Nix exists because it sends phony bug reports their way.

[nrabulinski]

Sounds like the problem is with Python maintainers who don’t understand that breaking changes should only be made between major versions.

If that’s not possible though then as sibling comment said - you can override the dependencies and the nix maintainer should make sure the package works as expected

[hfkwer]

Sounds like the problem could also be with Nix maintainers who don't understand that "semver" is not a universal law of nature and that not all projects and ecosystems follow it. This kind of blanket dismissal can cut both ways.

Semver (the website and "spec") was created in 2009 by some guy. It's not an RFC, a standard, or anything like that. Yes, it gained widespread adoption. Yes, the guy in question is a cofounder of GitHub. So what? You cannot force it upon everyone. Python is about 20 years older than semver. Django is several years older. Should the whole ecosystem change their conventions because it's more convenient for a few people?

[nrabulinski]

Except Django site says that a.b are feature releases which should be backwards compatible except for specific exceptions. If their software truly breaks “with every update to django_4” then it’s either a problem on Django’s side or a problem in how said person uses Django

[hfkwer]

I don't know if it's deliberate or a communication/comprehension problem, but you're misquoting Django's release process https://docs.djangoproject.com/en/dev/internals/release-proc...

> * Versions are numbered in the form A.B or A.B.C.

> * A.B is the feature release version number. Each version will be mostly backwards compatible with the previous release. Exceptions to this rule will be listed in the release notes.

> * C is the patch release version number, which is incremented for bugfix and security releases. These releases will be 100% backwards-compatible with the previous patch release. The only exception is when a security or data loss issue can’t be fixed without breaking backwards-compatibility. If this happens, the release notes will provide detailed upgrade instructions.

Going from "mostly backwards compatible with the previous release. Exceptions to this rule will be listed" to "should be backwards compatible except for specific exceptions" is quite the stretch. There are no "specific exceptions": incompatibilities can be anywhere and you need to read the release notes to know where. In semver, a minor version increment is backwards-compatible, no exception, no ifs or buts.

If you want to shoehorn Django's release process into "semver", then act as if the product is called "Django 4". If the version is "Django v4.X.Y", then X is the major version number, Y is the minor version number, and there is no patch version. It should be version in Nix as "django4 vX.Y.0".

[dlahoda]

i use nix for 2 years. never was thinkin or relied on semver. exactly nix allows me not to consider semver as something realy existing or working.

[lolinder]

Doesn't this auto-upgrade behavior punch straight through the reproducibility Nix is supposed to be giving you? It's not exactly a functional build system if the results you get depend on when you download the dependencies.

(I mean, I guess you could say that time is an input to the function, but that seems to miss the point.)

[hamandcheese]

What do you mean by auto-upgrade behavior?

If the Django package in nix were upgraded, all packages that use it would be tested.

And you wouldn't get the upgrade automatically, instead you would only get the upgrade when you change the version of Nixpkgs that you are using.

And if you don't like that, then you can use multiple versions of Nixpkgs at the same time. Your old package will stay exactly as it was. This of course cuts both ways, and means you get no security updates for it or any of its transitive dependencies.

Which part of this isn't reproducible or functional? If nixpkgs never changed, it wouldn't be a very good package repository.

[Cu3PO42]

Using Flakes, you can lock the version of nixpkgs (and any other repository) to a certain commit, and that commit is an input to the function. When you update that commit, of course the build changes, but I'd say that's pretty expected. If you don't upgrade it, you'll keep the prior versions.

Now this only works as long as you keep your package outside of he main nixpkgs repository, once you upstream it you're locked into the versions of packages that are "currently" in nixpkgs in the same commit. Builds are still reproducible, because you select the commit you build, but your package might break if a dependency changes in an incompatible way. If that happens, there's a problem with either the definition of the application or the dependency. In the given case it sounds like there might be an issue with the package of the application since it seems it doesn't lock down the precise version of Django that it needs.

[assbuttbuttass]

You can think of the function inputs as:

1. All the package definitions in nixpkgs

2. Any external sources

When a package is updated in nixpkgs, input #1 changes.

[lolinder]

I mean, I get that, but that means that the reproducibility of my build depends on the whims of the nixpkgs maintainers, it's not a property guaranteed by the package manager.

[tymscar]

You can however define inputs that are not the whole of nixpkgs. You would use something like this and you would pin it to a very exact version and hash of a package:

https://nixos.org/guides/nix-pills/nixpkgs-overriding-packag...

[pxc]

The goal of a downstream Linux distribution is never to reproduce whatever builds you run on your own machine as an upstream developer. It's to produce a collection of installable software that meets various constraints and goals, like cohesion (can all be installed and managed uniformly), minimal size, easy/manageable security updates, integration (compatibility and so on). That can involve things like building the software against particular library versions mandated by downstream needs or even patching it. Some distros try hard to avoid patching upstream and some don't, and in all distros there may be cases where other priorities take precedence over the value of leaving upstream untouched.

In the case of Nixpkgs and Python, the community wants to maintain a collection of Python libraries that are all interoperable, and Python doesn't support vendorization well enough to allow multiple versions of the same library in a single Python process, which is one reason for preferring singular versions of most Python libraries in Nixpkgs. The other factor is likely just reducing the maintenance across Nixpkgs by maintaining as few redundant versions within the tree as possible.

If you want to control/determine the entire runtime your end users use, you have to do the packaging work required to ship them that runtime with some tooling that's capable of the reproducibility you desire. Python doesn't have one a reproducible package manager, so your options are basically creating your own Nix package (probably as a flake.nix in your repo), Docker, and Flatpak.

That said, it's perfectly possibly to include multiple minor releases of Django 4 in a single snapshot of the Nixpkgs tree and maybe that should be done. Have you talked with the maintainers of your downstream package of Nixpkgs to let them know Django breaks things on minor releases, and so using different versions of Django 4 interchangeably is not tested or supported in your application?

[ptman]

You can pin the commit of nixpkgs. Why should reproducibility hold when nixpkgs changes?

[midchildan]

I think there's a bit of confusion caused by equating Nix "derivations" with "packages" of traditional package managers.

Nix mainly concerns itself with derivations [1]. They're build recipes for creating binary artifacts that are meant to be consumed by the Nix daemon. The Nix daemon instantiates derivations by building the artifact and storing it to a store path under /nix/store. Store paths are unique to each derivation.

When people say Nix is reproducible, they mean that derivations are reproducible [2]. This is because anything that might cause the build to change is captured as inputs to the derivation. Every input is explicitly specified by the author of the derivation. This means that when a dependency gets updated, the resulting derivation and store path would change. The new derivation might fail to build, but the old one would still continue to build regardless of how much time has passed since it was first built. So if a latest package in Nixpkgs is broken, you can always go back to a known good commit to get a working derivation while waiting for the package maintainer to fix it [3].

Traditional package managers don't have a concept of a derivation. Instead, they have packages. Those packages have no reproducibility whatsoever. Even if they built successfully in the past, they might not build today. That's because a traditional package is only identified by its name and version, as opposed to a Nix derivation which is identified by its content (= the build recipe) [4]. Traditional package managers see two incompatible builds with the same name and version as the same package, replaceable with each other. Worse, most package managers don't require versions to be specified as part of dependencies. Whether a package builds or not is then dependent on the current state of the central package repository. Again, this isn't the case with Nix derivations.

[1]: Internally, Nix doesn't even have the concept of a package. A package is a concept that we humans use to group related derivations together.

[2]: To be clear, derivations aren't bit-by-bit reproducible. For example, CPU caches would be observable during builds because in general, process sandboxes don't prevent hardware information leakage. However, it's reproducible in a practical sense because people would have to go out of their way to make software builds dependent on things like CPU state. People might do that as a joke, but not for any serious reason.

[3]: Ideally, tests and reviews should catch any breakage but sometimes it happens. Hence the rolling release branch is marked "unstable." Fortunately, it's also easy to apply fixes locally before they're available in Nixpkgs because Nix makes it straightforward to create a custom derivation by extending existing ones.

[4]: Not to be confused with content addressed derivations, which identifies derivations by the resulting binary artifact.

[dlahoda]

for 2 there is sandbox from facebook to isolate tests (and builds) from cpu non determinism. i have raised ticket on nix. so really it is just another derivation sandbox.

[pxc]

> It is definitely possible with Nix to use the precise versions of what's in your requirements.txt, but I'm not sure if the Nixpkgs maintainers would allow all that extra duplication upstream.

They do for end user applications, but not for Python libraries. The libraries in Nixpkgs are expected to be interoperable, which requires converging certain versions because otherwise transitive dependencies on varying library versions mean that libraries used together are subject to serious, mysterious bugs. But applications packaged in Nixpkgs can pull in an exact set of libraries of their own if that's what it takes for them to run reliably.

[nurple]

It sounds like the package is implemented improperly. If the input from your repo to the package is not targetting a specific commit, it should be.

Building from "latest" is really not how nix is ever meant to operate. In that case, when you update your requirements.txt, it is now out of sync with the package definition; the inputs _have_ changed and your guarantees are gone.

When your project repo is updated, that should never result in a change to what gets installed by nixpkgs until you also update the package to point at that commit and do any work necessary to fix breaking changes. Once you do that work, that version of your package picks up a guarantee to always be producable.

Like another comment mentioned, this is all much easier to accomplish with flakes as they have a lockfile that sits next to the flake, both of which reside in your repo and can be updated atomically with your releases instead of also needing to make a PR for nixpkgs.

I've actually been working on learning how to better package python with nix and found the historical information on python packaging infrastructure in this talk incredibly enlightening (I think this landed on HN a few days back): https://www.youtube.com/watch?v=ADSM4vR2EQ0

[jacereda]

The answer today would probably be to use flakes: https://nixos.wiki/wiki/Flakes

[SuperSandro2000]

They don't need to use the same version of the Django package but Python dependency pins are often either way to tight and can easily be expanded or outright missing, so they often get ignored.

[pxc]

Are you opposed to filing a bug in Nixpkgs for your application? Alternatively, are you willing to point to your application or its package in Nixpkgs so that someone else can do so?

[goodpoint]

> Of course we, like all other python projects, don't support using other dependency versions then the ones in the requirements.txt file.

That's really bad. You should always support reasonable version ranges.

> when someone just uses a different minor version of django, stuff breaks

That's why some people say that managing dependencies in Python is difficult and move to statically compiled languages.

[SuperSandro2000]

> That's why some people say that managing dependencies in Python is difficult and move to statically compiled languages.

Yes, completely agreeing with that.

[pxc]

It's not even really about static compilation. NixOS (and many Linux distros) include tons of dynamically linked C applications that just do a way, way better job of compatibility. Imagine if GNU grep were as fussy about only being built against 1 version of glibc as many Python libraries and applications seem to be about their dependencies.

[goodpoint]

I don't.

[lolinder]

What makes you think that the versions specified in the requirements.txt aren't reasonable ranges? All OP is saying is that if you're outside the version ranges in requirements.txt then you're outside the supported range. It's literally in the name of the file—requirements.

[goodpoint]

> What makes you think that the versions specified in the requirements.txt aren't reasonable ranges?

Because that's what the parent wrote.

[dlahoda]

for rust or haskell i point nix to my version and lock and toolchain files. and it uses exact tools and versions default build uses, but built with nix.

hope such solution exists for python.

[otabdeveloper4]

nixpkgs doesn't use requirements.txt for whatever reason.

(That reason probably being the utter brokenness and braindead state of Python packaging; Node packages work much better.)

[awegio]

> Node packages work much better

Are you sure about that? I haven't seen a node app built from source on nixpkgs yet. That includes Electron apps like Signal Desktop, which is a bit disappointing.

There is this article about trying to package jQuery on Guix:

http://dustycloud.org/blog/javascript-packaging-dystopia/

[misterio7]

Grep nixpkgs for `buildNpmPackage`, it's ridiculously easy to package a node app nowadays.

[otabdeveloper4]

Yes, buildNpmPackage works great.

[rekado]

Guix has several different npm importers (none of them merged), but it's debatable whether it is desirable to build npm packages from source when it either creates thousands of barely useful packages.

[the_gipsy]

You can package simple python projects, but as soon as there are too many huge deoendecies that use CPython and whatnot, it becomes impossible to generate the nix derivation. I just use imperative python-venv + pip install on those.

[lloeki]

Take a look at Home Assistant to see a complex python app being packaged in Nix.

(disclaimer: it's still rough but it does work)

https://github.com/NixOS/nixpkgs/tree/master/pkgs/servers/ho...

[otabdeveloper4]

It doesn't, but you need to ditch requirements.txt and just overridePythonPackage with the correct github revision hash.

It's a PITA but unlike pip and conda it's 100% reliable.

[SuperSandro2000]

> I just use imperative python-venv + pip install on those.

The whole point of NixOS is to manage this and to get rid of those manual steps that are error prone.

[the_gipsy]

I only do this on some work project that I don't touch often, as an escape hatch. Everything else is managed by nix.

[goodpoint]

Wasn't Nix supposed to solve these problems?

[kaba0]

It does. Nix can package everything properly. What is depending on the language ecosystem in question is whether this packaging can be more automized or not.

Python is not trivially automatized with Nix.

[Filligree]

And it does, for most languages. Python seems more difficult than average.

[the_gipsy]

Yea python is the exception. Go, rust, nodejs, have been easy to get running with specific versions and dev envs .

[goodpoint]

...and yet there's tons of Python packaged in traditional distributions including Django.

Nix promises to solve exactly this problem... so it's not clear what the real benefit of Nix is.

EDIT: a rain of silent downvotes?

[globular-toast]

> Of course we, like all other python projects, don't support using other dependency versions then the ones in the requirements.txt file. So when someone just uses a different minor version of django, stuff breaks

That sounds wrong. A Python package should not have a requirements.txt file at all. A requirements.txt file is for "freezing" and fully reproducing an environment (ie. in a virtualenv or docker container). This is useful for certain applications like deploying services or sharing notebooks etc. It is not for packages. A package should document its requirements via setup.py/pyproject.toml and do so in the loosest way possible. Django uses semver and Django apps don't generally need to pin to minor versions.

Stuff like this is why people think Python packaging is worse than it really is.

[scandinavian]

The application is not distributed via pypi, nor is it installed as a package and thus have no setup.py file.

> A requirements.txt file is for "freezing" and fully reproducing an environment (ie. in a virtualenv or docker container).

No, it's just for specifying which versions of packages should be installed by pip. There's no such concept of a lock file with pip. Poetry and the likes have lock files though.

[duped]

> There's no such concept of a lock file with pip.

There's the --require-hashes flag and the ability to specify the hashes in your requirements.txt

[SAI_Peregrinus]

Django doesn't use semver. It uses a Major.Feature.Patch release notation, not Major.Minor.Patch. Feature releases usually contain breaking changes, where SemVer minor releases never should.

[reuben364]

With my ignorance of the python packaging ecosystem, I was always under the impression that requirements.txt was the version constraints, not the lock file.