[wavemode]

Probably not. NPM has its problems but Python packaging has always been significantly messier (partly because, Python is much older than Node and, indeed, much older than the very concept of resolving dependencies over the internet).

[oblio]

The bigger problem in Python has been its slowness and reliance on C dependencies.

Maven solved Java packaging circa 2005, for example. Yes, XML is verbose, but it's an implementation detail. Python still lags on many fronts, 20 years later.

An example: even now it makes 0 sense to me why virtual envs are not designed and supposed to be portable between machines with the same architecture (!). Or why venvs need to be activated with shell-variety specific code.

[zahlman]

> An example:

None of this example has anything to do with performance or reliance on C dependencies, but ok.

> even now it makes 0 sense to me why virtual envs are not designed and supposed to be portable between machines with the same architecture (!).

They aren't designed to be relocatable at all - and that's the only actual stumbling block to it. (They may even contain activation scripts for other platforms!)

That's because a bunch of stuff in there specifies absolute paths. In particular, installers (Pip, at least) will generate wrapper scripts that specify absolute paths. This is so that you can copy them out of the environment and have them work. Yes, people really do use that workflow (especially on Windows, where symlinking isn't straightforward).

It absolutely could be made to work - probably fairly easily, and there have been calls to sacrifice that workflow to make it work. It's also entirely possible to do a bit of surgery on a relocated venv and make it work again. I've done it a few times.

The third-party `virtualenv` also offers some support for this. Their documentation says there are some issues with this. I'm pretty sure they're mainly talking about that wrapper-script-copying use case.

> Or why venvs need to be activated with shell-variety specific code.

The activation sets environment variables for the current shell. That isn't possible (at least in a cross-platform way) from Python since the Python process would be a child of that shell. (This is also why you have to e.g. use `source` explicitly to run the Linux versions.)

But venvs generally don't need to be activated at all. The only things the activation script effectively does:

* Set the path environment variable so that the virtual environment's Python (or symlink thereto) will be found first.

* Put some fancy stuff in the prompt so that you can feel like you're "in" the virtual environment (a luxury, not at all required).

* Set `VIRTUAL_ENV`, which some Python code might care about (but they could equally well check things like `sys.executable`)

* Unset (and remember) `PYTHONHOME` (which is a hack that hardly anyone has a good use case for anyway)

* (on some systems that don't have a separate explicit deactivate script) set up the means to undo all those changes

The actually important thing is the path variable change, and even then you don't need that unless the code is going to e.g. start a Python subprocess and ask the system to find Python. (Or, much more commonly, because you have a `#!/usr/bin/env python` shebang somewhere.) You can just run the virtual environment's Python directly.

In particular, you don't have to activate the virtual environment in order to use its wrapper scripts, as long as you can find them. And, in fact, Pipx depends on this.

[oblio]

> None of this example has anything to do with performance or reliance on C dependencies, but ok.

<C dependencies>

You'd realize why I wrote that if you used Java/Maven. Java is by and large self-contained. Stuff like Python, Ruby, PHP, Javascript[1] etc, are not, they depend on system libraries.

So when you install something on Solaris, FreeBSD, MacOS, Windows, well, then you have to deal with the whole mess.

1. Is the C dependency installed on the system? 2. Is it the correct major version or minor version? 3. Has it been compiled with the correct flags or whatnot? 4. If it's not on the system, can the programming language specific package manager pull a binary from a repo for my OS-arch combo? 5. If there's no binary, can the programming language specific package manager pull the sources and compile them for my OS-arch combo?

All of those steps can and do fail, take time, and sometimes you have to handle them yourself because of bugs.

Java is fast enough that almost everything can be written in Java, so 99% of the libraries you use only have 1 artifact: the universal jar, and that's available in Maven repos. No faffing around with wheels or whatnot, or worse, with actual system dependencies that are implicitly (or explicitly) required by dependencies written in the higher level programming language.

<Virtual envs>

I won't even bother to address in detail the insanity that you described about virtual envs, it's just Stockholm syndrome. Almost every other programming language does just fine without venvs. Also I don't really buy that issue with the lack of portability, it's just a bunch of bad design decision early on in Python's design. Even for Windows there are better possibilities (symlinks are feasible, you just need admin access).

I say this as someone who's used basically all mainstream programming language over the years.

The sane way to do virtual envs is to have them be just... folders. No absolute paths, no activation, just have a folder and a configuration file. The launcher automatically detects a configuration file with the default name and the configuration file in turn points the launcher and Python to use the stuff in the folder.

Deployment then becomes... drumroll just copying the folder to another machine (usually zipping/unzipping it first).

* * *

[1] Javascript is a bit of a special case but it's still slower than Java, on average.

[zahlman]

> Stuff like Python, Ruby, PHP, Javascript[1] etc, are not, they depend on system libraries.

The Python runtime itself may depend on system libraries.

Python packages usually include their own bundled compiled code (either directly in the wheel, or else building an sdist will build complete libraries that end up in site-packages rather than just wrappers). A wheel for NumPy will deliver Python code that uses an included copy of OpenBLAS, even if your system already had a BLAS implementation installed.

Regardless, that has no bearing on how virtual environments work.

> I won't even bother to address in detail the insanity that you described about virtual envs, it's just Stockholm syndrome. Almost every other programming language does just fine without venvs.

You say this like you think there's something difficult or onerous about using virtual environments. There really isn't.

> Also I don't really buy that issue with the lack of portability

The use of absolute paths is the only thing preventing you from relocating venvs, including moving them to another machine on the same architecture. I know because I have done the "surgery" to relocate them.

They really do have the reason for using absolute paths that I cited. Here's someone from the Pip team saying as much a few days ago: https://discuss.python.org/t/_/96177/3 (Using a relative path in the wrapper script would of course also require a little more work to make it relative to the script rather than to the current working directory. It's worse for the activation script; I don't even know if that can be done in pure sh when the script is being sourced.)

Yes, it's different between Linux and Windows, because installers will create actual .exe wrappers (stub executables that read their own file name and then `CreateProcess` a Python process) instead of Python wrapper scripts with a shebang. They do this explicitly because of the UX that Windows users expect.

> Even for Windows there are better possibilities (symlinks are feasible, you just need admin access).

Please go survey all the people you know who write code on Windows and see how many of them have ever heard of a symlink or understand what they are. But also, giving admin rights to these Python tools all the time is annoying, and bad security practice (since they don't actually need to modify any system files).

> The sane way to do virtual envs is to have them be just... folders. No absolute paths, no activation, just have a folder and a configuration file.

A virtual environment is just a folder (hierarchy) and a configuration file. Like I said, activation is not required to use them. And the fact that certain standard tools create and expect absolute paths is not essential to what a virtual environment is. If you go in and replace the absolute paths with relative paths, you still have a virtual environment, and it still works as you'd expect - minus the tradeoffs inherent to relative paths. And like I said, there is third-party tooling that will do this for you.

Oh, I guess you object because there are actual symlinks (or copies by default on Windows) to Python in there. That's the neat thing: because you start Python from that path, you don't need a launcher. (And you also, again, don't need to activate. But you can if you want, for a UX that some prefer.) Unless by "launcher" you meant the wrapper script? Those are written to invoke the venv's Python - again, you don't need to activate. Which is why this can work:

  sudo apt install pipx
  pipx install sphinx
  sphinx-quickstart

A venv is created, and used without activation.

Again, yes, the wrapper scripts would have to be a little more complex in order to make relative paths work reliably - but that's a Pip issue, not a venv issue.

[oblio]

> But also, giving admin rights to these Python tools all the time is annoying, and bad security practice (since they don't actually need to modify any system files).

You only need to do it once, when the symlink is created...

My point is that venvs are a code smell. Again, there's a reason basically no other programming language ecosystem needs them. They're there because in Python's history, that was the best idea they had 20 years ago (or whenever they were created), where they didn't even bother to do their homework and see what other ecosystems did to solve that specific problem.

> You say this like you think there's something difficult or onerous about using virtual environments. There really isn't.

They're a bad, leaky, abstraction. They provide value through a mechanism that is cumbersome and doesn't even work well compared to basically all competing solution used outside of Python.

* * *

Anyway, I've used Python for long enough and I've seem many variations of your argument often enough that I'm just... bored. Python packaging is a cluster** and it has sooo many braindead ideas (like setup.py having arbitrary code in it :-| ) that yes, with a ton of work invested in it, if you squint enough, it basically works.

But that doesn't excuse the ton of bad design around it.

[zahlman]

> Again, there's a reason basically no other programming language ecosystem needs them.

You have failed to explain how they are meaningfully different from what other programming language ecosystems use to create isolated environments, such that I should actually care about the difference.

Again: activating the venv is not in any way necessary to use it. The only relevant components are some empty folders, `pyvenv.cfg` and a symlink to the Python executable (Windows' nonsense notwithstanding).

> 20 years ago (or whenever they were created), where they didn't even bother to do their homework and see what other ecosystems did to solve that specific problem.

Feel free to state your knowledge of what other ecosystems did to solve those problems at the time, and explain what is substantively different from what a venv does and why it's better to do it that way.

No, a .jar is not comparable, because either it vendors its dependencies or you have to explain what classpath to use. And good luck if you want to have multiple versions of Java installed and have the .jar use the right one.

> They're a bad, leaky, abstraction. They provide value through a mechanism that is cumbersome and doesn't even work well compared to basically all competing solution used outside of Python.

You have failed to demonstrate this, and it does not match my experience.

> it has sooo many braindead ideas (like setup.py having arbitrary code in it :-| )

You know that setup.py is not required for packaging pure-Python projects, and hasn't been for many years? And that it only appears in a source distribution and not in wheels? And that in other ecosystems where packages contain arbitrary code in arbitrary foreign languages that needs to be built on the end user's machine, the package also includes arbitrary code that gets run at install time in order to orchestrate that build process? (For that matter, Linux system packages do this; see e.g. https://askubuntu.com/questions/62534 .)

Yes, using arbitrary code to specify metadata was braindead. Which is why pyproject.toml exists. And do keep in mind that the old way was conceived of in a fundamentally different era.

[int_19h]

The upside in Python is that dependencies tend to be more coarse grained and things break less when you update. With JS you have to be on the treadmill constantly to avoid bitrot, and because packages tend to be so small and dependency trees so large, there's a lot of potential points of failure when updating anything.