[bborud]

Python has other, bigger problems that make it a constant headache. One of them being the dismissive attitude towards any and all of problems that come from versioning, dependencies and quirks that make it challenging to have robustness.

Criticisms are typically dismissed by suggesting heaping yet another "solution" onto the growing pile of "solutions" that you have to drag around with you. That people have to learn. That you have to install tooling for. That has to be vetted. That has to become part of the toolbox to get even seemingly simple things done. This attitude is a big part of the reason that I strongly advise people against using Python in production. On top of all the problems presented in a real-world setting.

Almost all of the time, people who are fond of Python are more interested in defending python, disparaging me, downvoting me etc that listen to why I make that recommendation.

(I get it. People like Python. What I think of Python as a language is irrelevant. In fact I don't have that much against it. But I do have a lot against it in a setting where you need reliability and repeatability)

I have spent the last month of my life building a system that can run Python tooling reliably in a business critical application. I knew this was going to be a pretty big job when I started, but for every problem I solve, a bunch of new problems arise. I am starting to see light at the end of the tunnel but it hasn't exactly been smooth sailing. I'm almost there for a first version, but there are a bunch of problems still to solve. Mostly because I care about developer ergonomics and that things should "just work". One important goal is that my solution shouldn't impose any significant cognitive burden on people who use it. That's really hard.

(I don't think the solution will be open source since my contract wouldn't allow for it. But I'll make the case at some point for why it should be open sourced)

And yes. There are statically typed languages available today that have decent tooling that provides superior developer ergonomics. I can understand that people don't want to learn new languages, but if you have the capacity to do so I would recommend trying to move on if the code you write has to run outside your own workstation. If an old fart like me can learn and adopt new languages, so can you.

[chuckadams]

Python the language is pretty nice. It has its warts, but I make my living in PHP which is practically made of warts. But the python ecosystem still seems to be trying to figure out this whole package management and project setup thing. In most languages I can do some form of `$blub install` where $blub is the language's official package manager or some close equivalent. It's just python that always screams at me that I have to set up and "enter" a virtualenv. I get what venv is for, but it's still a weird hack of hardlinks and relative paths that no other language seems to need, and a clumsy two-step dance of a UX that hasn't improved in like 20 years.

[necovek]

I am not sure you get what virtualenvs are: Python is never screaming at you to set up a virtualenv, it must be a particular package recommending use of virtualenv for easy set up without interfering with the rest of your system.

Virtualenv allows you to seamlessly run multiple Python ecosystems simultaneously, even within the same project directory. It's basically primitive containerisation mechanism that predates any actual containerisation systems on Linux.

You do not have to use it, but then you can easily slip into a sort of "DLL hell" (multiple incompatible library versions installed system-wide) with multiple projects — or need to bundle all dependencies within your project directly. None of this is specific to Python, really — any shared library system has the same challenge. How many other systems are there in active use making it as easy to use multiple incompatible versions of shared libraries per project or within the same project?

When in doubt, you can always retreat to the basics in Python world: put packages you need in a path of your choice, and point PYTHONPATH (sys.path) at it.

[chuckadams]

I thought I'd given sufficient clues that I actually do in fact know what virtualenv is. I was hip-deep into python when it was introduced, and thought it was a clever hack, but it's relying on python being compiled to find its libraries in a path relative to the python binary, so rather than use something sensible like a launcher that sets PYTHONPATH to a local dir like every other language does, it hardlinks the python install into the local dir in order to pretend everything is a system-wide global install. I always figured that hack was a temporary workaround, but it's going on a couple decades now.

As for the screaming, I'm talking about pip, which has the unique property of just refusing to work by default unless you use --user (what most other package managers now call "global") or are in one of these pretend-global environments. Ultimately it's just a paper cut, not a show-stopper, but it points to not just a failure to address the pain points of the package ecosystem, but a seeming refusal to do so.

[necovek]

Usually running pip as root overcame that problem — but it did cause all the other problems virtualenvs were introduced to work around (so I am not suggesting this as the solution).

You are right that there are many problems with packaging in Python, and yet I feel like virtualenvs are the smallest of those.

I believe we also need to compare tech this old to tech from the same era: obviously newer ecosystems had the benefit of hindsight, but how does managing dependencies compare between Perl and Python, for instance?

The biggest problem with Python packaging is — IMO at least — that it is actually attracting so many evolutions and proposals that ot is hard to stay on top of if you do not make Python packaging your core interest.

[chuckadams]

For managing dependencies in Perl, it was originally a similar story to Python: everything was system-installed, but many people would install things to their home dir and set PERL5LIB in their .bashrc. The cpan client was smart enough to detect and use the home install when writing its initial config, so you could call it a day. Later there was local::lib which fiddled @INC for the use of a project-local directory, and cpanminus defaulted to using it, and then Carton came around which is more or less a clone of Bundler from Ruby, also using local::lib under the covers.

[nijave]

What are the problems you have with tooling? Imo it's no worse than most other languages besides a very small handful of recent ones (rust, go) where everything is included

The easy approach is usually just throw it in an OCI container

There's not much concrete to go on here besides "I don't like the ergonomics"

[bborud]

Throwing it in an OCI container is not the "easy" approach. It is the beginning of opening a can of worms. I know, because I've been developing tooling to do that at scale for the last few months and behind every layer of worms there are uglier and bigger worms that you need to deal with.

And yeah, I have written a lot of code to insulate Python in containers while allowing meaningful access to hardware and services. While at the same time not heaping more complexity and cognitive surface at the developer. Including writing my own container software to actually understand what's involved at a more detailed level so I know what I'm doing when trying to make this work with existing container software. (No, I don't run any of my container managers in production since I don't want to maintain it -- but this also means a bit more complexity in using existing ones)

It may be "easy" in trivial cases, but it is very far from easy if you want to make something that can cater to a wide range of scenarios.

[nijave]

Do you have examples of issues? OCI is tarball of dependencies that doesn't fight with the OS userland.

Why do you need to write code to insulate Python in containers?

At the simplest level, you can add the flags to the container runtime (network host, host ipc, host process namespace) to turn off all the namespacing besides filesystem and the Python container runs just like a non containerized process.

An extreme example

https://github.com/home-assistant/docker-base (Debian, Ubuntu, Alpine base images with Python for arm64 and amd64)

& https://github.com/home-assistant/core/blob/dev/Dockerfile (Python app built on those with >1000 deps)

And even there most of the custom code is just running a ton of combinations of inputs against docker build. The OCI container gets rid of "wide range of scenarios" for you standardizing the runtime environment