Hacker News new | ask | show | jobs
by brettcannon 2264 days ago
Do note that Python the language is developed by a separate group as Python the packaging ecosystem. So even when Guido was a BDFL he completely stayed out of the packaging situation.

And there is a packaging BDFL, but there's a severe lack of time and effort in the form of volunteers to tackle a lot of the projects, big or small. Plus rolling these things out takes years and it takes even longer to gain traction as people are often reluctant to change their build tools once things work.

As an example of timelines, take what my blog post covers. PEP 518 and the concept of pyproject.toml was conceived in 2016, four years ago. Getting PEP 517 finalized and it all worked into pip didn't happen until pip 19 in January 2019, over a year ago (https://pip.pypa.io/en/stable/news/#id203). And a year later there's still enough of a need to educate folks that I wrote this blog post to help get the word out.

IOW the timelines are huge and the pay-off is way down the line. And that's if you manage to be motivated enough to do the work. Burn-out is a huge problem in the packaging world. I mean reading that you think "it is a fucking mess and it's unacceptable" doesn't motivate me and others to keep trying to improve things if that's how us volunteers will be treated for years to come while we try to rectify the situation. Add on to the fact that we aren't getting paid for this and it makes the idea of pulling out my laptop tonight to work on some packaging stuff feel rather pointless.

So please be understanding that people are working on things as best they can and that overt negativity about it all will work against you getting what you want.

3 comments

Definitely, it is always good to keep in the back of our minds that FOSS is developed by people in their free time and on a volunteering basis. No one owes us anything. No one should feel entitled. I am sorry if my complains come out as such. Python runs the world and it is because of the contributions of many amazing people.
> Definitely, it is always good to keep in the back of our minds that FOSS is developed by people in their free time and on a volunteering basis.

Please don't over-generalize. Volunteering is not the only way FOSS is developed. E.g. the Linux kernel, Gitlab, Firefox, Golang main tools…

This is BS. most other languages, with the same constraints, have done far better. Take Ruby as one example.
Ruby serves one ecosystem (web), barely.

Python serves several huge ones: web, datascience, 3d graphics, sysadmin... all with different needs, different conventions, different “standard” tools to do this and that. The community is orders of magnitude bigger than ruby by now, and anything that requires coordination is much harder to accomplish than in a lot of other ecosystems.

Python is now on par with ecosystems like C/C++ (where different build and packaging conventions are legions) or Java (where ever 10 years they write a new build tool on top of the previous one(s) to hide the complexity, and still, like python, they struggle to make simple executables 30 years on...). It’s just a hard problem to cater for so many different needs.

That’s the “oh we can’t use methods from others, we are too special” argument, which is always BS. Coupling the misapprehension to a snooty misconception is doubly unfortunate.
> Python serves several huge ones: web, datascience, 3d graphics, sysadmin... all with different needs, different conventions, different “standard” tools to do this and that.

Them having different conventions and standard tools is the failure, not a constraint.

What are the different needs that these ecosystems have which mean they couldn't use a common package manager? I don't believe there are any.

For instance, in the scientific computing world, compiled extensions are hugely important, whereas they're relatively rare in web development.

There's probably no logical reason why a single common package manager for these domains is impossible. But they have different priorities, and people write tools that solve their own problems. Who is going to decide which package managers are unnecessary, and convince all of the users who are happy with them to switch to something else?

NPM has excellent support for compiled extensions, despite being mostly used for web development.
I think something people are forgetting when comparing Python's packaging system to other languages and runtimes is how old Python and its ecosystem is. For instance, Python's public release predates Linux's public release(Feb 1991 versus Aug 1991). The first release of setuptools was 2004 and it isn't like people weren't sharing Python code even before then (https://github.com/pypa/setuptools/commit/8423e1ed14ac1691c2...). That's over 5 years before npm was released (https://en.wikipedia.org/wiki/Npm_(software)). That means npm got to learn from Python and any other project that predated it. It also means they didn't have Python's momentum to contend with (I certainly never expected Python to become _this_ popular, and that makes it very hard to change as you literally have tens of millions of people to re-educate when you change something).

So while it's fine to say you wished Python's packaging approach was as good as another languages, please do so know that it is not fair to say that if one community got something to work for their needs it means Python should have been able to as well; it's not an apples-to-apples comparison considering the trees were quite possibly planted at different times and are different varietals.

> relatively rare in web development

not at all. from my previous experience with ruby i was using gems with compiled bindings to libraries for: mysql, memcached, json, xml, regexp and http. and probably some more i don't recall right now.

Python does all of that and more. There are still occasional issues with the thorniest setups, but binary wheels absolutely took care of the average case. You even get a massive desktop toolkit luke QT with a simple “pip install pyqt”.

Getting there might not be the nicest experience, but most packages nowadays can and do get there.

Working in 3D graphics there are lots of different constraints unfortunately and it isn't an easy solution.

A lot of 3d graphics houses have to roll their own solution.

This is a famous one in the cycle and a common approach-> https://github.com/nerdvegas/rez

The issue is that the environment varies a lot and there's usually a lot of C++ code that needs to be compiled to interact with the python plugins based on different environments with 500+ libraries mixing & matching...

Node does it better as well. So does java with maven which has been the dominant way to do it in java for over a decade.
I would not use Node as an example of good packaging. It really hasn't been that long since left-pad. Then there's the fact that you still have multiple different tools to do the same thing (npm vs. yarn) and the end result node_modules/ is a fucking mess anyways.

    find node_modules/ -name "node_modules" | wc -l
         327
I think you can certainly use Node as an example of much better developer experience. Even the difference between npm and yarn is fairly minimal, and there is a solid community 'default' in Yarn.

I think part of the reason we ended up with so many trivial libraries and dependencies in the Node ecosystem was precisely because it is so easy to create and publish packages.

I love Python as a language, but working with the packaging even just to install dependencies is like pulling teeth every time you start a new project - nevermind publishing packages yourself.

No it's not... 99.9% of the time, installing dependencies is a breeze. Packaging is where the issue lies, but pulling those dependencies down is painless. You run one command and everything works. Unlike say npm/node/yarn where 50% of the time after coming back to work on a project your node_modules is broken and you have to either reinstall npm or node or node + npm.
Node's packaging story sucks in many ways, but IMO python's is worse. I spent quite some time working with python when there was no first-party dependency sandboxing (i.e. there was virtualenv but no venv) and even today there are tools that work with one but not the other.

That said, PEPs 517 and 518 sound like they're going in the right direction from the article.

ied ( http://gugel.io/ied/ ) had a great idea for reducing redundant installations in node_modules/, I wish the project got some traction. It worked as advertised when I used it 4 years ago.
That’s a different issue than the one we’re talking about here.
Maven is “so good” that people wrote another tool on top (Gradle) so they don’t have to touch the Mavenstrosity. More or less like Maven was built to get rid of that other nightmare, Ant.
Maven isn't as good as bundler or npm for sure, but it's better than the mess that is the python communities solution.
left-pad comes to mind when you mention Node.
Yes, Python’s packaging is partly bad because it’s based on C extensions, which are intrinsically bad, but it’s also just plain bad compared to its peers, like Ruby and NPM.
NPM also has support for C extensions, and they mostly work reliably (the only problems I've had are forgetting to `npm rebuild` when switching between node versions, or when switching between native mac builds and linux-in-docker builds.
What exactly is "intrinsically bad" about extensions written in C?
In my personal experience, extensions written in C don't necessarily compile at first attempt because the dependency management there is even worse. May be it fails with a missing openssl or readline library and I have to go find whatever OS specific thingy I need to do to recover.
C/C++ have no equivalent of npm/bundler/pip. Instead there are incompatible flavors of Makefiles for incompatible C compilers. It's a hard problem space to work in.
A lot of ruby is based on C extensions too.
It might be said that python solved a LOT of packaging woes by including so many batteries.

Compared to other languages, python has a pretty rich set of modular tools in the standard library.