[colemannerd]

BUT importantly... it REALLY does not work well for lots of teams. For me, this setup has caused production outages multiple times across multiple teams. Maybe the root python ecosystem should learn and adopt from other ecosystems that have figured out complex deployment in a much easier way.

[yjftsjthsd-h]

> For me, this setup has caused production outages multiple times across multiple teams

How does that happen?

[MrJohz]

I've seen people use that setup but not freeze the dependencies and so have errors in production that didn't exist in development and waste days trying to figure out what was going on.

I've seen people use this setup and then struggle to deploy in different environments, especially when a dependency updates and no longer works correctly on a particular device, or where there are differences in behaviour or packaging or something in two different machines.

I've seen people accidentally install packages locally and not add them to the requirements file (especially when they're less experienced with Python), and cause outages by having the application crash on startup.

I've seen people freeze the dependency list and then have excess dependencies floating around because they couldn't differentiate between dependencies that were being used, and dependencies that were previously transitively installed and no longer needed. This doesn't necessarily cause outages, but does slow everything down over time, either in continual package maintenance or in downloading excess packages.

Most of the time, when I've seen teams use this sort of "simple" packaging process, they end up writing a bunch of scripts to facilitate it (because it's rarely so simple in practice). I have seen these scripts fail in almost every possible way. Often this happens in a development environment or before production, but I've seen production issues here as well.

To be clear, I think there are some situations where .venv and requirements.txt really are all you need. But I don't think going down that route removes complexity or makes things easier. Instead, it means you need to manage that essential complexity yourself. There are sometimes advantages to that, and reasons why it might make sense to take that option, but they are relatively rare. And given that pip/venv are right now the most official way of handling packaging in Python, that raises a massive red flag for the entire ecosystem.

[colemannerd]

I've personally seen the issues the article mentions with pyenv and virtual envs first hand when relying on pip.

[yjftsjthsd-h]

The problem mentioned with pyenv is that people accidentally develop/test on the wrong version of python itself. But that's specific to pyenv, and I don't actually see where the article discusses problems with venv. So again: What exact steps would a team take using just pip+virtualenv or pip+conda (the comment you responded to didn't mention pyenv or venv) that would lead to production outages?

[colemannerd]

It feels like you've determined there's nothing wrong with pyenv, pip, and virtualenv so any issues brought up, you will reject.

If that's not the case, here's the issue - someone used pyenv and did not exactly specify the python type - I believe we were on 3.9 and prod was 3.9.11 and the current python version was 3.9.12. There was a downstream package that had an OS dependency - I believe it was pandas - that conflicted and needed to be updated locally to work with 3.9.12. This broke and raised an error in production that was not reproducible locally - and when you deploy on AWS, reproducing can be a pain in the butt. I'm sure if the data scientist had used perfect pyenv, virtualenv, and pip commands; we would have caught this. However, they're very complicated - especially for people who focus on math - so requiring full knowledge of these tools is unrealistic for most data scientists.

[yjftsjthsd-h]

> It feels like you've determined there's nothing wrong with pyenv, pip, and virtualenv so any issues brought up, you will reject.

Alternatively, I'm rejecting your claims because you keep making them and then not providing evidence. Now that you've actually described the problem, I can agree that that's a footgun, and pyenv should start to strongly discourage setting a global version in much the same way that pip has started to protect against people using `sudo pip install` to trash their systems.