Hacker News new | ask | show | jobs
by pkage 707 days ago
ML researcher perspective: Conda is... dog slow, even for relatively simple tasks (clone and run a project). The recommendation nowadays is to use Mamba (iirc), but in my experience (a few years old now) it's been unstable and unable to find package solves which worked on my system / our cluster.

I've settled on just using Poetry for most things, and then using pip in a venv to install the pyproject.toml file from Poetry either in a Dockerfile or directly on a cluster. That's worked fairly well so far, even with torch/cuda (and the mess of CUDA versioning) and from macOS to Linux.

I think uv/rye is a good next step, Poetry can be a bit slow as well at times.

3 comments

It IS slow, no argument there, but I never find the speed of a package management tool too important.

Maybe it's different for other ecosystem such as node etc., but when I'm doing research in ML I config my project mostly just once and do the bulk work (install cuda pytorch etc.), later it's mostly just activate and occasionally add some util packages via pip.

What makes conda better than native venv+pip is its extensive libraries/channel and be able to solve/build complicated dependencies effortlessly especially when you have to run your project on both Windows and Linux.

This is not to say speeding up isn't needed, of course!

> What makes conda better than native venv+pip is its extensive libraries/channel and be able to solve/build complicated dependencies effortlessly especially when you have to run your project on both Windows and Linux.

For me, most stuff is installed via pip anyways. The only things I'm pulling via conda is blas, scipy, torch and all that stuff that's a PITA to install.

If you are working on a large collaborative project, switching between branches can mean needing to rebuild your container images. It's not something I do every day, but it happens enough that the difference between 1 minute (doesn't disrupt flow/train of thought) and 10 minutes (disrupts flow) means something.
The mamba solver comes with conda nowadays. It's not slow any more.
Not only it is slow, it has so many idempotency issues that it makes it barely usable.