[dalemyers]

This isn't quite the same. Yes, I can update my root dependencies in the `requirements.txt` file, run `pip install -r requirements.txt` and the run `pip freeze > requirements.txt`, but that's convoluted and requires me to know exactly what my root dependencies are. Is `astroid` something our tools use directly, or is it just a dependency of `pylint`? It's not clear. A lockfile clears this up.

[ptx]

Yes, and in addition to the requirements file, pip supports a constraints file, which is the lockfile you describe. It's separate from the requirements file. It solves exactly this problem.

Docs: https://pip.pypa.io/en/stable/user_guide/#constraints-files

[nickjj]

The docs for this mentions:

> Including a package in a constraints file does not trigger installation of the package.

Maybe I'm not following something but how do you get all of this to work like a lock file in other package managers?

Let's use Ruby as a working example:

1. You start a new project and you have a Gemfile.

2. This Gemfile is where you define your top level dependencies very much like a requirements.txt file. You can choose to version lock these dependencies if you'd like (it's a best practice), but that's optional.

3. You run `bundle install`

4. All of your dependencies get resolved and installed

5. A new Gemfile.lock file was created automatically for you. This is machine generated and contains a list of all dependencies (top level and every dependency of every dependency) along with locking them to their exact patch versions at the point of running step 3.

6. The next time you run `bundle install` it will detect that a Gemfile.lock file exists and use that to figure out what to install

7. If you change your Gemfile and run `bundle install` again, a new Gemfile.lock will be generated

8. You commit both the Gemfile and Gemfile.lock to version control and git push it up

At this point you're safe. If another developer clones your repo or CI runs today or 3 months from now everyone will get the same exact versions of everything you had at the time of pushing it.

[ptx]

It should be the same process, except that the constraints file is not automatically created or detected, so step 5 would be "pip freeze >constraints.txt" and step 6 would be "pip install -r requirements.txt -c constraints.txt".

The top level dependencies go in requirements.txt and trigger installation of those packages. Everything else goes in the constraints file, which constrains the version that will be installed if something triggers an installation of the package, but it doesn't by itself trigger the installation - it only locks/constrains the versions.

[nickjj]

Wouldn't you also need to run a pip3 install -r requirements.txt before the pip freeze?

Otherwise pip freeze won't find any dependencies.

So you end up having to run something like this:

    pip3 install -r requirements.txt
    pip3 freeze > requirements-lock.txt
    pip3 install -r requirements.txt -c requirements-lock.txt

Mainly because you can't run pip3 install -c requirements-lock.txt on its own it seems. It requires the -r flag.

That is a lot more inconvenient than running `bundle install` and if you use Docker it gets a lot more tricky because a new lock file would get generated on every build which kind of defeats the purpose of it, because ideally you'd want to use the existing lock file in version control, not generate a new one every time you build your image.

[jtdev]

Nice! I’ll be adding this to my virtualenv + requirements.txt + pip process. Not sure why everyone wants to overcomplicate Python dependency management with pyenv/poetry/etc.

[jshen]

How do you validate versions against the constraint file? I read your original link a few times and didn’t see it.

[ptx]

pip install -r requirements.txt -c constraints.txt

You can also, thanks to the weird way requirements.txt works, put the line "-c constraints.txt" in requirements.txt. In that case you don't have to specify it when you run pip.

That should apply the constraints when installing packages. I don't know if there's also a way to validate what's already installed.