[arthur-st]

Any ongoing work/plans to follow for cross-platform lock files?

This is one concern that would prevent the team I'm on from moving. We do ML work, so I'll use a pytorch-based project for my example. The desired state is that all dependencies are in pyproject.toml, e.g., from which you can generate a set of lockfiles from an AArch64 Mac or an AMD64 Windows workstation for the following platform configurations:

1. Mac with default (MPS) pytorch

2. Windows with CUDA pytorch

3. AArch64 Linux with CPU pytorch

4. AMD64 Linux with CPU pytorch

5. AMD64 Linux with CUDA pytorch

P.S. Big thanks for making Ruff, we're huge fans of linting and formatting both.

[ewolf]

I have almost the same problem. With Poetry, I managed to work around it using this technique, involving a combination of groups and extras: https://github.com/lucaspar/poetry-torch

It's a hacky workaround, but it seems to work so far. It would be much nicer to see this solved in a better way, though!

[LarsDu88]

Poetry does cross platform lockfiles, but the absence of cross platform lockfiles is one of the reasons why uv benchmarks significantly faster (on top of using Rust for the resolve)

[arthur-st]

Unfortunately, installing pytorch via Poetry was not for the faint of heart, last I tried. [1]

For example, it used to download wheels for all matching distributions (say pytorch for a few platforms) to get metadata. That would be like a 10 GB download for 4 wheels of pytorch, to generate the lockfile.

Another thing it used to do is ignore local cache, so for some locking operations would download the same wheel multiple times.

In general, it was not entertaining, from an ML practitioner's perspective, to infer an attitude from the Poetry maintainers that the mountain shall come to Muhammad, so to say. Their abridged position was that pytorch is not compliant with standards, when Poetry itself, e.g., is not PEP 440-compliant, in a manner pertinent to the issue at hand. [2]

I realise that my grumbling about pytorch has nothing to do with your perfectly adequate comment, that said. Please interpret it as nothing more than an expanded rationale fore being more interested in learning about Rye/uv than in revisiting Poetry, at least for the projects I've worked on.

[1] https://github.com/python-poetry/poetry/issues/6409

[2] https://github.com/python-poetry/poetry/issues/7748

[LarsDu88]

My experience is that both PyTorch and Tensorflow are incredibly fickle with the interplay between cuda versioning and python package versioning.

You need to go straight into their websites and manually figure out what goes with what.

Quite honestly, this needs to be done in Docker with fancy conditional logic, and not via the python package manager.

[arthur-st]

Speaking about strictly PyTorch, it's actually elementary on the distribution side. I know that there are packages which do some weird setup magic at runtime, unfortunately, but pytorch is not one of them.

Installing it via the PyPI gets you the CPU version on Windows/macOS, and CUDA 12.1 on Linux. You can specify CUDA version explicitly, which you need to get CUDA on Windows, or CUDA 11.8 on Linux, by referencing the corresponding PyTorch package index, e.g., https://download.pytorch.org/whl/cu118 - cu118 means CUDA 11.8. Doing that explicitly is the normal practice, since in ML land you'll know your hardware well - no one is asking for tooling that automatically resolves CUDA/cuDNN/etcetra compatibility.

Hence, it hopefully follows that installing this with pip is trivial, you just need to maintain multiple requirements.txt files so that you specify the correct index URL where it's necessary. What I specifically want, then, is for someone to give me a UX where I don't have to maintain these requirements.txt files. I should be able to just specify arbitrary build environments* in pyproject.toml, and override the global default package index for specific packages in [platform-]specific build environments.

*Really, we're probably talking about combining blocks of optional dependencies into an ersatz build environment, but that's what they ultimately are for.

If I can then "compile" these build environments into properly reproducible lock files, and then use the dependency manager to export these lock files into conventional requirements.txt files with exact pins, then we've arrived at what I would call a good UX for specifying dependencies in Python.

To the best of my knowledge, for 99.9% of packages this doesn't require Docker. Different parts of this process work well across the ecosystem, just not as a matter of full package. To name a few

- Poetry has many building blocks, but just has had a questionable implementation as per parent you're replying to, and also random other problems like not working at all with private indexes back when I tried, and so on;

- pip-tools does the entire pipeline well, but it quite unfortunately doesn't support cross-platform targeting;

So, quite honestly, until someone can do this correctly and in one place - and, ideally, with endorsements from PSF and PyPA - I'll just consider Python dependency management tooling to be inadequate for my use cases.

[arthur-st]

I guess I'll add that this is mostly possible using a Docker-based workflow, but the hope is that one day we could get something more lightweight.

[HelloNurse]

Requiring Docker implies that a packaging tool is too simplistic to do its job.

[arthur-st]

Well, that's my question - I'm not familiar with Rye/uv, and I'm curious to know if it can substantively simplify the workflows that I'm aware of. I simply clarified to pre-empt a potential comment about Docker.