It is entirely possible to use poetry to determine the precise set of packages to install and write a requirements.txt, and then shotgun install those packages in parallel. I used a stupidly simple fish shell for loop that ran every requirements line as a pip install with an “&” to background the job and a “wait” after the loop. (iirc) Could use xargs or parallel too.
This is possible at least. Maybe it breaks in some circumstances but I haven’t hit it.
Not as an excuse for bad behavior but rather to consider infrastructure and expectations:
The packages might be cached locally.
There might be many servers – a CDN and/or mirrors.
Each server might have connection limits.
(The machine downloading the packages miiiiiight be able to serve as a mirror for others.)
If these are true, then it’s altruistically self-interested for everyone that the downloader gets all the packages as quickly as possible to be able to get stuff done.
I don’t know if they are true. I’d hope that local caching, CDNs and mirrors as well as reasonable connection limits were a self-evident and obviously minimal requirement for package distribution in something as arguably nation-sized as Python.
Can you recommend any good article / comparison of uv vs poetry vs conda?
We've used different combinations of pipx+lockfiles or poetry, which has been so far OK'ish. But recently discovered uv and are wondering about existing experience so far across the industry.
From my experience, uv is way better and it's also PEP compliant in terms of pyproject.toml. Which means in cas uv isn't a big player in the future, migrating away isn't too difficult.
At the same time, poetry still uses a custom format and is pretty slow.
How is uv so much faster? My understanding is Poetry is slow sometimes because PyPi doesn't have all the metadata required to solve things, so it needs to download packages and then figure it out.
If I recall correctly, uv is doing some ninja stuff like guessing the part of the relevant file that is likely to contain the metadata it needs and then doing a range request to avoid downloading the whole file.
It is entirely possible to use poetry to determine the precise set of packages to install and write a requirements.txt, and then shotgun install those packages in parallel. I used a stupidly simple fish shell for loop that ran every requirements line as a pip install with an “&” to background the job and a “wait” after the loop. (iirc) Could use xargs or parallel too.
This is possible at least. Maybe it breaks in some circumstances but I haven’t hit it.