Hacker News new | ask | show | jobs
by Borealid 546 days ago
That doesn't work well (enough) if you have one project that requires Python <3.10 and another that requires Python >=3.10.

To really pin everything you'd need to use something like asdf, on top of poetry or a manual virtualenv.

Otherwise you get your colleagues complaining that pip install failed with mysterious errors.

5 comments

venvs are namespace isolation, they are like containers.

Even in huge monorepos you can just use something like a Makefile to produce a local venv using PHONY and add it to clean too

This is how I actually test old versions of python, with versioned build targets, cython vs ...

You can set up almost any IDE to activate them automatically too.

The way to get you coworkers to quit complaining is to automate the build env setup, not fighting dependency hell, which is a battle you will never win.

It really is one of the most expensive types of coupling.

Would recommend you to install pyenv[1]. It was very useful when my team had to update a lot of projects using <=3.10 to 3.11 [1] https://github.com/pyenv/pyenv
I have multiple versions of Python built from source. If I want to test what my code will do on a given version, I spin up a new venv (near instantaneous using `--without-pip`, which I achieve via a small Bash wrapper) and try installing it (using the `--python` option to Pip, through another wrapper, allowing me to reuse a single global copy).

No matter what tooling you have, that kind of test is really the only way to be sure anyway.

If something doesn't work, I can play around with dependency versions and/or do the appropriate research to figure out what's required for a given Python version, then give the necessary hints in my `pyproject.toml` (https://packaging.python.org/en/latest/specifications/pyproj...) as environment markers on my dependency strings (https://peps.python.org/pep-0508/#environment-markers).

"Mysterious errors" in this area are usually only mysterious to end users.

I don't get it, you ought to be building a different venv per project anyway.

(Of course, I don't distribute most of my projects, so I just dump them all in the global install and don't worry about it)

Some libraries break across different versions of Python for a variety of reasons.

Pinning python version with asdf (in conjunction with a venv) gets you reasonably far in ensuring a certain project works across a lot of people in a team.

Venvs solve that problem, they don't cause it.

You activate the venv per project to get out of the global namespace problem.

As python x.nn versions are major with the nn part, it is like trying to mix c11 with c19.... things will break if you don't respect that.

A venv does not actually install a different Python interpreter. It's bound to the Python version that created it. You cannot make a Python 2.7 venv using a Python 3 interpreter. You need Python 3.10 to create a Python 3.10 venv.

There are plenty of situations where the Python interpreter version matters. As a non-exhaustive list, you have libraries that compile code, non-Python languages that link to cpython, build scripts that do different things depending on wheel/setup/other-bundled-stuff, Python code that uses removed compat shims like importlib-metadata...

If you haven't run into one of those situations yet, congratulations. I've been through this already, and making a reproducible environment does require first installing a pinned version of the Python interpreter and THEN setting up a venv using that particular interpreter.

> You need Python 3.10 to create a Python 3.10 venv.

Yes, but this doesn't need to cause a problem for those of us using bare-bones tooling. Speaking for myself, I just... run venv with that version of Python. Because I have a global installation of everything from 3.5 to 3.13 inclusive, plus 2.7 (just so I can verify what I post online about how things used to work in 2.x). And I never install anything to my base Python versions, because my system Python is Apt's territory and the ones I built from source are only and specifically there to make venvs with.

Compiling Python from source on Linux is about as straightforward as compiling from source ever gets, honestly. (The Python dev guide has some useful tips - https://devguide.python.org/getting-started/setup-building/i... .)

This seems like a solved problem with pyenv which is very popular. You can also include a `.python-version` in your git repo root dir to automatically use the correct python interpreter version when in the scope of that repo.
Yes. It's a solved problem for Python with pyenv. Until you want to pin a particular version of Poetry.

It's also a solved problem for Ruby with rvm, node with nvm, java with the JDK_HOME env var, Maven with mvnvm, ...

Or it's a solved problem for most every tool, programming language or not, with asdf.

Most OS's use package managers these days, expecting ANY language to not run into major problems with that is just asking for trouble.

Remember that Version SAT is NP-complete: https://research.swtch.com/version-sat

How do you expect to support violating that contract, where the OS vendor, package maintainers issue updates and security fixes, while you maintain your code, with the realization that Python releases have two years of bug fixes and three additional years of security fixes. Are you actually forced into this model or are you just adding to code debt refers and adding to the accumulation of poorly written or unmaintainable code that makes future changes even harder than they are today?

Using OS package managers helps you avoid accumulating so much code debt, making moving to newer versions the easy path actually results in better written and maintainable code.

If you just listen to depreciation warnings, and prioritize trying to stay close to the current released versions, and insisting that running on unsupported versions is an incident and not hiding it under the rug, things get better over time.

That said, the Python direct download paths are very stable.

3.13.0 being: www.python.org/downloads/release/python-3130/ 2.5.5 being: www.python.org/downloads/release/python-255/

I would still recommend creating packages from those downloads, while there are possibly better options, fpm has treated me well for well over a decade.

Looking in homebrew, which is probably one of the bigger unknowns, they have targets for everything going back to 3.8, with all major Linux distros supporting farther back and obviously VM/containers are an option there too.

  brew info --json python3 | jq -r '.[].versioned_formulae[]'
  python@3.12
  python@3.11
  python@3.10
  python@3.9
  python@3.8
I have run into the situations above, but the use of unsupported and/or end-of-life software is not something you intentionally help an organization do.

Especially with the actions by several agencies to bust EULA's

While IANAL we will see what happens in the courts, when the CISA labels something as "This dangerous practice is especially egregious in technologies accessible from the Internet"

There is a very big difference between sustainment needs and active development, one is a reality, the other could possibly be framed as "Gross negligence", invalidating any EULA protections in many parts of the country.

Obviously if you are running other peoples software the calculus of trade-offs changes.

But I have done the Python 2.7 to 3.x migration for several open source and commercial projects in the past. I can promise you that the time you waste on using 2.7 with software you control is far more expensive than updating.

If you are stuck on 2.7 on a product under active development, which has been EOL for 5 years and was warned to be going away for almost a decade prior to that...the problem you have is not with any 3rd party dependancies, the problem is with your organization.

No 3rd party vendor solution will fix those problems for you.

pyproject.toml solves this nowadays