[lr1970]

> I always see this and I wonder what other people are doing wrong. pip and a requirements.txt has worked for over a decade (along with virtualenv.)

Because python is just a tip of the iceberg. Python in Machine Learning is a glue and API that ties together multiple high performance numerical libraries and frameworks for CPU, GPU. Now you are fighting with your CUDA libraries, pytorch and its toolkits versions, BLAS, MKL, LAPACK, versions of you NVIDIA cards, support for non-standard floating point types (fp16,fp8), you name it. It is a zoo out there, python is just an obvious folly boy.

[icedchai]

I can believe that. No Python (or other language) dependency solution will be able to solve that. You need OS level configuration management. The average dev just installs and upgrades things willy-nilly.

[eternalban]

Open question here is whether a central repository at OS level solves anything. Logically, modulo reliable networking and fat pipes, the dependency could be coming from anywhere. So centralization doesn't appear to be the issue. The issue is the degree of version explosion for nominal dependency D by applications and figuring out acceptable transitive relations (alternative version) for imprecise matches. (And of course D will have its own set of dependencies.) If you can 'can' that, a systemic way to declare (exact), find (best attempt) match, and use (compatible) dependencies, you can serve up those dependencies from the network or canonical source or a local store cache. And only then you have solved the problem.

[icedchai]

Probably not. You need someone actually vetting dependencies, developing setup / onboarding scripts that install actual, approved, verified-working dependencies. OS packages are held, frozen at a specific version. Third-party installs are the same. You don't upgrade random stuff.

Obviously this is a lot less "agile" than most of us are used to.