[These335]

Can somebody please eli5 why it is so unanimously accepted that Python's package management is terrible? For personal projects venv + requirements.txt has never caused problems for me. For work projects we use poetry because of an assumption that we would need something better but I remain unconvinced (nothing was causing a problem for that decision to be made).

[tasuki]

In your requirements.txt, do you pin the concrete versions or leave some leeway?

If you aren't precise, you're gonna get different versions of your dependencies on different machines. Oops.

Pinning concrete versions is of course better, but then there isn't a clear and easy way to upgrade all dependencies and check whether ci still passes.

[marcosdumay]

You should use freeze files. Whatever language you are using, you should specify your dependencies on the loosest way possible, and use freeze files to pin them down.

The only difference from one language to another is that some make this mandatory, while in others it's only something that you should really do and there isn't any other real option you should consider.

[marcosdumay]

> For personal projects venv + requirements.txt has never caused problems for me.

That means you don't use Windows.

What is great. Keep not using it. But most people will have a different experience.

[kristianp]

Wait, are most of the people having problems in this thread using windows? It's been mentioned a couple of times, but not by most.

[disgruntledphd2]

There's a bunch of people who tend to have problems with Python dep management: - Windows users

- DS/compiled libs users (mostly Fortran/Cuda/C++)

- Anyone with dependencies on native/non python libraries.

Conda definitely helps with 2 and 3 above, and uv is at least a nice, fast API over pip (which is better since it started doing dependency checking and binary wheels).

More generally, lots of the issues come from the nature of python as a glue language over compiled libraries, which is a relatively harder problem in general.

[Kwpolska]

There are no Windows-specific issues in venv + pip. Windows can be more painful if you need to compile C extensions, but you usually don't, since most commonly used packages have had binary wheels for Windows on PyPI for many years.

[marcosdumay]

Installing anything in Windows is a system-specific problem.

If a pypi package has binary dependencies, it's a pip problem on Windows, if it depends on APIs that aren't installed by default, it's a pip problem on windows, if it depends on specific versions of something that isn't in python, it's a pip problem on Windows. If it depends on some API that the implementation is provided for free, but not freely, and is full of constrains on how you can install, it's a pip problem on Windows.

Most other package managers have the exact same problems. But some of the python alternatives that people often don't understand the point were created exactly to solve those problems.

[Kwpolska]

For using packages, venv + requirements.txt works, but is a bit clunky and confusing. Virtual environments are very easy to break by moving them or by updating your OS (and getting a new Python with it). Poetry is one alternative, but there are far too many options and choices to make. For building packages, there are similarly many competing options with different qualities and issues.

[rbanffy]

> Virtual environments are very easy to break by moving them

Virtual environments are cattle. Don’t treat them as pets. Just use pip freeze and recreate them elsewhere.

[shoo]

i think there might be merit to gdiamos's point that python is a popular language with a large number of users, and this might mean that python package management isn't unusually bad, but more users implies more complaints.

i think there was a significant step change improvement in python packaging around 2012, when the wheel format was introduced, which standardised distributing prebuilt platform-specific binary packages. for packages with gnarly native library dependencies / build toolchains (e.g. typical C/fortran numeric or scientific library wrapped in a layer of python bindings), once someone sets up a build server to bake wheels for target platforms, it becomes very easy to pip install them without dragging in that project's native build-from-source toolchain.

venv + pip (+ perhaps maintaining a stack of pre-built wheels for your target platform, for a commercial project where you want to be able to reproduce builds) gets most of the job done, and those ingredients have been in place for over 10 years.

around the time wheel was introduced, i was working at a company that shipped desktop software to windows machines, we used python for some of the application components. between venv + pip + wheels, it was OK.

where there were rough edges were things like: we have a dep on python wrapper library pywhatever, which requires a native library libwhatever.dll built from the c++ whatever project to be installed -- but libwhatever.dll has nothing to do with python, maybe its maintainers kindly provide an msi installer, so if you install it into a machine, it gets installed into the windows system folder, so venv isn't able to manage it & offer isolation if you need to install multiple versions for different projects / product lines, as venv only manages python packages, not arbitrary library dependencies from other ecosystems

but it's a bit much blame python for such difficulties: if you have a python library that has a native dependency on something that isnt a python package, you need to do something else to manage that dep. that's life. if you're trying to do it on windows, which doesn't have an O/S level package manager.. well, that's life.

[mardifoufs]

Try building a package and you will get hundreds of little paper cuts. Need a different index for some packages? It will work with a cli "pip install -from-index", but pip will not let you add an index in a requirement.txt for... security reasons. That means, good luck trying to "enforce" the CUDA version of pytorch without using third party tooling. So you either hard code a direct link (breaks platform portability), as that will work, or give up trying to make your project installable with "Pip install " Or "python build". Remember, pytorch basically has no CUDA builds anymore in its pypi index and no way to get CUDA torch from there (but I think this might have changed recently? )

Oh, and if some package you are using has a bug or something that requires you to vendor it in your repo, well then good luck because again, PEP 508 does not support installing another package from a relative link. You either need to put all the code inside the same package, vendored dependency included, and do some weird stuff to make sure that the module you wanted to vendor is used first, or... you just have to use the broken package, again for some sort of security reasons apparently.

Again, all of that might even work when using pip from the cli, but good luck trying to make a requirements.txt or define dependencies in a standard way that is even slightly outside of a certain workflow.

[vel0city]

You can include command line parameters like index-url and include-extra-index-url and find-links and what not in a requirements file.

[mardifoufs]

And have them build with "Pip install ." or python build? With the default setuptools config (or even any tweaks to it)? It works until you actually try to package the app, that's where the edge cases start piling up, a lot of them due to very weird decisions made on a whim on some random discourse thread.

Adding index URLs is explicitly not supported in the requirements.txt in setuptools or the default python build tool.

[invaliduser]

venv + requirements.txt has worked for every single python project I made for the last 2 years (I'm new to python). Only issue I had was when using a newish python version and not having a specific library released yet for this new version, but downgrading python solved this.

Being new to the ecosystem I have no clue why people would use Conda and why it matters. I tried it, but was left bewildered, not understanding the benefits.

[dagw]

I have no clue why people would use Conda

The big thing to realise is that when Conda first was released it was the only packaging solution that truly treated Windows as a first class citizen and for a long time was really the only way to easily install python packages on Windows. This got it a huge following in the scientific community where many people don't have a solid programming/computer background and generally still ran Windows on their desktops.

Conda also not only manages your python interpreter and python libraries, it manages your entire dependency chain down to the C level in a cross platform way. If a python library is a wrapper around a C library then pip generally won't also install the C library, Conda (often) will. If you have two different projects that need two different versions of GDAL or one needs OpenBLAS and one that needs MKL, or two different versions of CUDA then Conda (attempts to) solve that in a way that transparently works on Windows, Linux and MacOS. Using venv + requirements.txt you're out of luck and will have to fall back on doing everything in its own docker container.

Conda lets you mix private and public repos as well as mirroring public packages on-perm in a transparent way much smoother than pip, and has tools for things like audit logging, find grained access control, package signing and centralised controls and policy management.

Conda also has support for managing multi-language projects. Does your python project need nodejs installed to build the front-end? Conda can also manage your nodejs install. Using R for some statistical analysis in some part of your data pipeline? Conda will mange your R install. Using a Java library for something? Conda will make sure everybody has the right version of Java installed.

Also, it at least used to be common for people writing numeric and scientific libraries to release Conda packages first and then only eventually publish on PyPi once the library was 'done' (which could very well be never). So if you wanted the latest cutting edge packages in many fields you needed Conda.

Now there are obviously a huge class a projects where none of these features are needed and mean nothing. If you don't need Conda, then Conda is no longer the best answer. But there are still a lot of niche things Conda still does better than any other tool.

[disgruntledphd2]

> it manages your entire dependency chain down to the C level in a cross platform way.

I love conda, but this isn't true. You need to opt-in to a bunch of optional compiler flags to get a portable yml file, and then it can often fail on different OS's/versions anyway.

I haven't done too much of this since 2021 (gave up and used containers instead) but it was a nightmare getting windows/mac builds to work correctly with conda back then.

[dagw]

it was a nightmare getting windows/mac builds to work correctly

I think both statements can be true. Yes getting cross platform windows/Mac/Linux builds to work using Conda could definitely be a nightmare as you say. At the same time it was still easier with Conda than any other tool I've tried.

[drwu]

I am using Conda to build binary modules for different Python versions.

As user of the modules, venv is sufficient.

[mrweasel]

I can sort of the the argument, if you really really need to lock down your dependencies to very specific version, which I don't recommend you do.

For development I use venv and pip, sometimes pyenv if I need a specific Python version. For production, I install Python packages with apt. The operating system can deal with upgrading minor library versions.

I really hate most other package managers, they are all to confusing and to hard to use. You need to remember to pull in library update, rebuild and release. Poetry sucks too, it's way to complicated to use.

The technical arguments against Python packages managers are completely valid, but when people bring up Maven, NPM or even Go as role models I check out. The ergonomics of those tools are worse than venv and pip. I also think that's why we put up with pip and venv, they are so much easier to use than the alternative (maybe excluding uv). If a project uses Poetry, I just know that I'm going to be spending half a day upgrading dependencies, because someone locked them down a year ago and there's now 15 security holes that needs to be plugged.

No, what Python needs is to pull in requests and a web framework into the standard library and then we can start build 50% of our projects without any dependencies at all. They could pull in Django, it only has two or three dependencies anyway.

[gdiamos]

It's popular enough that it causes pain for a lot of people.

Coming from C++, IMO, it is vastly better.

[dccsillag]

Well, yes, but that's an extremely low bar!!