[toyg]

Ruby serves one ecosystem (web), barely.

Python serves several huge ones: web, datascience, 3d graphics, sysadmin... all with different needs, different conventions, different “standard” tools to do this and that. The community is orders of magnitude bigger than ruby by now, and anything that requires coordination is much harder to accomplish than in a lot of other ecosystems.

Python is now on par with ecosystems like C/C++ (where different build and packaging conventions are legions) or Java (where ever 10 years they write a new build tool on top of the previous one(s) to hide the complexity, and still, like python, they struggle to make simple executables 30 years on...). It’s just a hard problem to cater for so many different needs.

[inopinatus]

That’s the “oh we can’t use methods from others, we are too special” argument, which is always BS. Coupling the misapprehension to a snooty misconception is doubly unfortunate.

[twic]

> Python serves several huge ones: web, datascience, 3d graphics, sysadmin... all with different needs, different conventions, different “standard” tools to do this and that.

Them having different conventions and standard tools is the failure, not a constraint.

What are the different needs that these ecosystems have which mean they couldn't use a common package manager? I don't believe there are any.

[takluyver]

For instance, in the scientific computing world, compiled extensions are hugely important, whereas they're relatively rare in web development.

There's probably no logical reason why a single common package manager for these domains is impossible. But they have different priorities, and people write tools that solve their own problems. Who is going to decide which package managers are unnecessary, and convince all of the users who are happy with them to switch to something else?

[nicoburns]

NPM has excellent support for compiled extensions, despite being mostly used for web development.

[brettcannon]

I think something people are forgetting when comparing Python's packaging system to other languages and runtimes is how old Python and its ecosystem is. For instance, Python's public release predates Linux's public release(Feb 1991 versus Aug 1991). The first release of setuptools was 2004 and it isn't like people weren't sharing Python code even before then (https://github.com/pypa/setuptools/commit/8423e1ed14ac1691c2...). That's over 5 years before npm was released (https://en.wikipedia.org/wiki/Npm_(software)). That means npm got to learn from Python and any other project that predated it. It also means they didn't have Python's momentum to contend with (I certainly never expected Python to become _this_ popular, and that makes it very hard to change as you literally have tens of millions of people to re-educate when you change something).

So while it's fine to say you wished Python's packaging approach was as good as another languages, please do so know that it is not fair to say that if one community got something to work for their needs it means Python should have been able to as well; it's not an apples-to-apples comparison considering the trees were quite possibly planted at different times and are different varietals.

[twic]

The age also means Python has had plenty of time to come up with something new!

[brettcannon]

I'm not sure how old you are, but if you don't know the name "Borland" you might not realize how novel it is to have a compiler on every major OS. While Python's age means it has been around to see a lot of changes occur, that also means it had pre-existing practices and code to also consider whenever a shift in the industry occurred. Plus in hindsight things look obvious, but during the transition it isn't obvious what will stick and what won't.

It's pretty remarkable that things have continued to function since my 17 years with Python alone have seen things like:

- Compilers becoming available on all OSs - Linux becoming a thing - Laptops becoming common - Smartphones - Web-based email clients

As I have said a few times in comments on this thread, it's fine to feel like there's room for improvement, but trying to "stick it" to Python and its packaging ecosystem in some way for not being perfect is pointless for you and demotivating for those who are actually in a position to try to make things better for you (and even if you aren't a Python user you should care to some extent about it as Instagram runs on Python and thus it is helping to deliver you those cat photos while you're stuck at home ).

[takluyver]

People in the Python ecosystem have come up with plenty of new things over the years - eggs, wheels, conda, poetry... Coming up with something new is the easy bit. Getting everyone to agree that a new thing is better and stop using/recommending other things is much harder - almost impossible, if those other things serve some use cases fairly well.

The fragmented ecosystem is one of the main complaints about Python packaging today. And that's really tricky, because the main way the open source world knows to solve problems is to create new tools, and that doesn't help when the problem is 'too many tools'.

[BerislavLopac]

And it (and by "it" I mean the Python community) absolutely did! The main problem that it has come up with numerous different somethings that have been competing to become standards, across a wide range of related, but separate areas (packaging, versioning, environments...), and only a few of them (like pip) managed to become partial (but not at all universal) de facto standards.

[keymone]

> relatively rare in web development

not at all. from my previous experience with ruby i was using gems with compiled bindings to libraries for: mysql, memcached, json, xml, regexp and http. and probably some more i don't recall right now.

[toyg]

Python does all of that and more. There are still occasional issues with the thorniest setups, but binary wheels absolutely took care of the average case. You even get a massive desktop toolkit luke QT with a simple “pip install pyqt”.

Getting there might not be the nicest experience, but most packages nowadays can and do get there.

[aprdm]

Working in 3D graphics there are lots of different constraints unfortunately and it isn't an easy solution.

A lot of 3d graphics houses have to roll their own solution.

This is a famous one in the cycle and a common approach-> https://github.com/nerdvegas/rez

The issue is that the environment varies a lot and there's usually a lot of C++ code that needs to be compiled to interact with the python plugins based on different environments with 500+ libraries mixing & matching...

[jshen]

Node does it better as well. So does java with maven which has been the dominant way to do it in java for over a decade.

[Hamuko]

I would not use Node as an example of good packaging. It really hasn't been that long since left-pad. Then there's the fact that you still have multiple different tools to do the same thing (npm vs. yarn) and the end result node_modules/ is a fucking mess anyways.

    find node_modules/ -name "node_modules" | wc -l
         327

[mjfisher]

I think you can certainly use Node as an example of much better developer experience. Even the difference between npm and yarn is fairly minimal, and there is a solid community 'default' in Yarn.

I think part of the reason we ended up with so many trivial libraries and dependencies in the Node ecosystem was precisely because it is so easy to create and publish packages.

I love Python as a language, but working with the packaging even just to install dependencies is like pulling teeth every time you start a new project - nevermind publishing packages yourself.

[zo1]

No it's not... 99.9% of the time, installing dependencies is a breeze. Packaging is where the issue lies, but pulling those dependencies down is painless. You run one command and everything works. Unlike say npm/node/yarn where 50% of the time after coming back to work on a project your node_modules is broken and you have to either reinstall npm or node or node + npm.

[jshen]

The problem we're talking about isn't "installing a dependency", the problem is creating reproducible builds across people and between dev and production. The secondary problem is managing dependencies over the long run (updating, transitive dependencies, etc). Ruby/Bundler does this so much better than requirements.txt by a long shot.

[zodiac]

Node's packaging story sucks in many ways, but IMO python's is worse. I spent quite some time working with python when there was no first-party dependency sandboxing (i.e. there was virtualenv but no venv) and even today there are tools that work with one but not the other.

That said, PEPs 517 and 518 sound like they're going in the right direction from the article.

[nurettin]

ied ( http://gugel.io/ied/ ) had a great idea for reducing redundant installations in node_modules/, I wish the project got some traction. It worked as advertised when I used it 4 years ago.

[jshen]

That’s a different issue than the one we’re talking about here.

[toyg]

Maven is “so good” that people wrote another tool on top (Gradle) so they don’t have to touch the Mavenstrosity. More or less like Maven was built to get rid of that other nightmare, Ant.

[jshen]

Maven isn't as good as bundler or npm for sure, but it's better than the mess that is the python communities solution.

[collyw]

left-pad comes to mind when you mention Node.