Hacker News new | ask | show | jobs
by a_cool_username 2210 days ago
Let me start by saying: I love python, and I love developing in it. It's the "a pleasure to have in class" of languages: phenomenal library support, not too painful to develop in, nice and lightweight so it's easy to throw together test scripts in the shell (contrast that with Java!), easy to specify simple dependencies + install them. (contrast that with C!).

That said... if you work on software that is distributed to less-technical users and have any number of dependencies, python package management is a nightmare. Specifying dependencies is just a minefield of bad results.

- If you specify a version that's too unbounded, users will often find themselves unable to install previous versions of your software with a simple `pip install foo==version`, because some dependency has revved in some incompatible way, or even worse specified a different dependency version that conflicts with another dependency. pip does a breadth-first search on dependencies and will happily resolve totally incompatible dependencies when a valid satisfying dependency exists.[1]

- If you specify a version with strict version bounds to avoid that problem, users will whine about not getting the newest version/conflicting packages that they also want to install. Obviously you just ignore them or explain it, but it's much more of a time sink than anyone wants.

- In theory you can use virtualenvs to solve that problem, but explaining how those work to a frustrated Windows user who just spent hours struggling to get Python installed and into their `PATH` is no fun for anyone. Python's made great strides here with their Windows installers, but it's frankly still amateur hour over there.

- Binary packages are hell. Wheels were supposed to make Conda obsolete but as a packager, it's no fun at all to have to build binary wheels for every Python version/OS/bitness combination. `manylinux` and the decline of 32-bit OSes has helped here, but it's still super painful. Having a hard time tracking down a Windows machine in your CI env that supports Python 3.9? Too bad, no wheels for them. When a user installs with the wrong version, Python spits out a big ugly error message about compilers because it found the sdist instead of a wheel. It's super easy as a maintainer to just make a mistake and not get a wheel uploaded and cut out some part of your user base from getting a valid update, and screw over everyone downstream.

- Heaven help you if you have to link with any C libraries you don't have control over and have shitty stability policies (looking at you, OpenSSL[2]). Users will experience your package breaking because of simple OS updates. Catalina made this about a million times worse on macos.

- Python has two setup libraries (`distutils` and `setuptools`) and on a project of any real complexity you'll find yourself importing both of them in your setup.py file. I guess I should be grateful it's just the two of them.

- Optional dependencies are very poorly implemented. It still isn't possible to say "users can opt-in to just a specific dependency, but by default get all options". This is such an obvious feature, instead you're supposed to write a post-install hook or something into distutils.

- Sometimes it feels like nobody in the python packaging ecosystem has ever written a project using PEP420 namespaces. It's been, what, 8 years now? and we're just starting to get real support. Ridiculous.

I could go on about this for days. Nothing makes me feel more like finding a new job in a language with a functioning dependency manager than finding out that someone updated a dependency's dependency's dependency and therefore I have to spend half my day tracking down obscure OS-specific build issues to add version bounds instead of adding actual features or fixing real bugs. I have to put tons of dependencies' dependencies into my package's setup.py, not because I care about the version, but because otherwise pip will just fuck it up every time for some percentage of my users.

[1] I am told that this is "in progress", and if you look at pip's codebase the current code is indeed in a folder marked "legacy".

[2] I 100% understand the OpenSSL team's opinion on this and as an open source maintainer I even support it to some degree, but man oh man is it a frustrating situation to be in from a user perspective. Similarly, as someone who cares about security, I understand Apple's perspective on the versioned dylib matter, but that doesn't make it suck any less to develop against.

3 comments

> struggling to get Python installed and into their `PATH` ... it's frankly still amateur hour over there

But that has been solved on Windows for quite a while hasn't it?

Python installs the "py" launcher on the path, which allows you to run whichever version you want of those you have installed. Just type "py" instead of "python". Or "py -3.5-32" to specifically run 32-bit Python 3.5, or "py -0" to list the available versions.

It's gotten a lot better, but we still hit tons of issues with users who don't know what Python version they installed their application in. Oh and of course our "binaries" in Scripts/bin don't seem to show up in the PATH by default. So I get to tell people "py -3.8-64 -m foo" on windows, "foo" everywhere else.

This gets much much worse when a new version of Python comes out and we don't support it yet (because of the build system issues I mentioned). I spent several weeks teaching people how to uninstall 3.8 and install 3.7 before we finally got a functioning package out for 3.8.

I like Mozilla build system on Windows, you click "start-shell.bat" and it runs console. Python, mercurial, rust - just works, never checked PATH.

https://firefox-source-docs.mozilla.org/setup/windows_build....

Sure, but telling people to run "py -3.7" seems a lot easier than walking them through uninstalling and reinstalling Python, as you would have had to in the bad old days. It's reliable and consistent and doesn't depend of what's installed where or how it's configured. If you run "py -3.7 -m venv my_env", it just works, always, with no special context required.

Although I don't handle user support for Python packages, if I did, that would be my go-to approach.

If only there was some graphical tool that allows the user to see conflicts, relax version dependencies, and of course rollback changes if things didn't work out.

Or an error message like:

    There's a version conflict. In order to resolve, try one of the following:

    pip relax-dep package1 >= 1.0
    pip relax-dep package2 >= 2.0
    pip remove package3
And then you would want to have

    pip undo
(Just brainstorming here.)
Looks like any other package manager:

* developers install with language packager

* in between install with OS package manager

* users install bundle

Those who have troubles with pip, gems, cabal, etc should check over options first.

Wait, bundlers Gemfile.lock lists installed versions at least ten years, what is "too unbounded" in pip?

It depends! Sometimes I have to lock a dependency at minor releases because every.single.release from the author breaks something new, and I've already worked around the locked version's failings. Sometimes I have to lock a dependency at a major version and everything is fine after that. Usually when the latter happens, eventually the developer releases something that fits within the version bounds and breaks. Sometimes they fix it in the next release, but then I have to deal with a week of bug reports from users that "I couldn't pip install the latest release!". A big complaint I'll with flask/werkzeug app is that something or other broke because they installed something else with strict version requirements alongside it (because the authors of that program have experienced the same bullshit, I assume).

Maybe I'm spoiled from working with cargo and npm (I have almost no ruby experience so I can't comment there), but both of them have way fewer such version conflicts in my experience. Obviously there are tradeoffs and I don't want the node_modules experience for my users, but often it seems that would be a much better experience than pip for everyone. With either of those, I just "npm install" or "cargo install" and all my dependencies end up there working.

You can generate a requirements.txt file using "pip freeze" on a functioning system, but then you have to figure out a way to point users at it instead of using "pip install myapp". Also you might have to do it for each OS since windows vs mac vs linux can have different package dependencies specified, and even if you don't do that, a dependency doing it means you have to account for it.

You can copy+paste the "pip freeze" output into your setup.py and add quotes+commas, but then you're back to breaking side-by-side packages.

So what am I, a developer trying to distribute my command-line application to less-technical users, supposed to do? Distribute two entirely different packages, "myapp-locked" and "myapp"? Tell people to install from a copy+pasted "requirements.txt" file? I've started distributing docker containers that have the application installed via the requirements.txt method, which is fucking stupid but at least the users of that complain less about versioning issues... until the day someone yanks a package I guess.

I've recently reported bug on Xmonad github, they have

### Checklist

  - [ ] I've read [CONTRIBUTING.md](https://github.com/xmonad/xmonad/blob/master/CONTRIBUTING.md)

  - [ ] I tested my configuration with [xmonad-testing](https://github.com/xmonad/xmonad-testing)
I think it is brilliant idea, immediately checked latest git versions, I assume you may add

  - [ ] I tested my application with [latest stable requirements.txt](...)
And something about triangulation and reporting to another repo too.

Sorry to hear about breaks on major version. Ruby gems (libraries) freeze dependencies on major, sometimes minor, example [0]. But applications shipped with Gemfile and Gemfile.lock [1], [2]. So `bundle install` is reproducible [3]:

> The presence of a `Gemfile.lock` in a gem's repository ensures that a fresh checkout of the repository uses the exact same set of dependencies every time. We believe this makes repositories more friendly towards new and existing contributors. Ideally, anyone should be able to clone the repo, run `bundle install`, and have passing tests. If you don't check in your `Gemfile.lock`, new contributors can get different versions of your dependencies, and run into failing tests that they don't know how to fix.

Yes, docker, msi, Flatpack, AppImage - whatever works for you and your users. It is sad we can't easily statically compile in one file on scripting languages.

[0] https://github.com/teamcapybara/capybara/blob/master/capybar...

[1] https://github.com/Shopify/example-ruby-app/blob/master/Gemf...

[2] https://github.com/Shopify/example-ruby-app/blob/master/Gemf...

[3] https://bundler.io/guides/faq.html