[PhilipRoman]

There seems to be a severe lack of hash pinning in "modern" software ecosystems. We figured out how to do it 20+ years ago with Git bringing hash addressed storage to the masses. Coming from a different background it was very surprising for me to see things like docker images, packages and github actions being updated at the whim of upstream registry. I much prefer the philosophy where builds are fully offline and predictable, even if not fully reproducible.

[woodruffw]

I’m not sure what this has to do with TFA: Python does have hash-pinning. TFA is not about modifying existing files on the index (PyPI doesn’t allow that), but about adding new files to a pre-existing release. But that doesn’t change the hash of older distributions on that release.

[nobodyandproud]

I feel you’re quibbling over semantics here.

In concept why can’t the full set of files in a release be a single, one-way hash value, with both adding or releasing changing the hash value?

[woodruffw]

Because uploads to PyPI are not atomic. They’re now capped within a 14 day window, but it would be extremely confusing to users to have the “release” hash of their dependencies change repeatedly.

(Also: we’d need to determine what it even means to hash a set of files. Do we order by canonicalized filename, by upload time, etc. Each of these has surprising implications!)

Edit: to be clear, it’s not ideal or fully intentional that uploads to PyPI are not atomic. But it’s the status quo and any change will take years.

[nobodyandproud]

> Because uploads to PyPI are not atomic. They’re now capped within a 14 day window, but it would be extremely confusing to users to have the “release” hash of their dependencies change repeatedly.

Aren’t releases versioned, using semantic versioning?

If not, why shouldn’t a change in dependencies not trigger a new release version?

If it is versioned and if there’s a corresponding hash to the release, why wouldn’t I also expect that to change when the version and dependencies change?

I can understand that there may be historical reasons that this scheme will break how PyPI releases are built and distributed, but I also hope you understand that it violates the principle of least surprise.

[woodruffw]

Python uses PEP 440 versioning, which isn’t semver. Semver is de facto common, however, and can be expressed as a subset of PEP 440.

I’m not sure what you mean by “change in dependencies”: every release has zero or more distributions (“files”), and each distribution in a release can specify its own dependencies. For example, a macOS-specific wheel might depend on something that Linux-specific wheels don’t need or vice versa.

> but I also hope you understand that it violates the principle of least surprise.

I think it’s fair to say that virtually everything about Python packaging violates POLA :-)

[nfbdhdfbf]

These things you’re posing as design questions are already solved 1 million times over. If you don’t want to design something new, just use Git semantics. These are not excuses for inaction. Python package management has been a horrifying embarrassment for 15 years and there’s no excuse for the amount it’s been neglected as every other ecosystem has shown better ways.

If it weren’t for AI, Python would have died as a language ecosystem due to their perpetual neglect of this area. The python community clearly had no ability to solve this problem themselves. I got sick and left and started using better tools.

It’s only now that they’re the cornerstone of the world economy that other people are coming in and cleaning up their messes for them. It’s pitiful. Python failed and needed to be saved from itself.

[joombaga]

The semantics are important. A release is composed of multiple distributions (sets of files). Each distribution does have a one-way hash value. This is what you lock to. Adding files does change the value of that hash. Files cannot be removed or changed.

You lock to the distribution instead of the release so your build doesn't have to download distributions for platforms you're not using.

[PunchyHamster]

Which is the entire problem. "Release" should be a signed hash of a manifest file that contains hashes of all the other files (in case of Git that's what commit does).

PyPi just decided to do what VCSes already did, worse

[nobodyandproud]

It’s not. Steel manning the OP (which at face value seems to be the most reasonable interpretation) they’re wondering why an entire release isn’t hash pinned.

The response is that there is hash pinning but only at the constituent levels. Which clearly isn’t what the OP meant.

[crabbone]

1. There are multiple levels at which things are hashed, signed and checked when it comes to Python packages. Eg. each file in the Wheel beside the RECORD file is hashed using SHA256. This hash is never checked :( You can also sign your packages (RECORD.jws anyone? Is that still supported?), but nobody checks that either.

2. There are hashes in the HTML served by PyPI. These are updated at the whim of both the index and the publisher. Even though they are checked by pip during install, they are worthless.

3. There are many ways to install packages that work around (2). Custom index server doesn't have to provide hashes, and pip will happily install that. You can install from sources, from a package you've downloaded somewhere, form VCS, you can build it during install, all without even prompting the user to confirm the very scary choices.

NB. I have no idea how do you make the leap from "adding files to release" to "not modifying the release". To me, adding file to release is sure as hell modifying it. Here's a very simple malicious example:

I release package "innocent" with an empty "scripts" section. Then, in the subsequent modification to this release, I add the "scripts" section with a script named "notebook". Now, whenever my user wants to run Jupyter notebook, they will call my "notebook" program, not the one from Jupyter package.

[woodruffw]

There are a lot of misconceptions here:

1. “Wheel” signatures (e.g. RECORD.jws) were specified but never actually implemented in any major packaging installer, to the best of my knowledge. In that sense it’s not really meaningful to ask whether they’re “still” supported, since nothing ever did support them unfortunately. The reasons for that are multifaceted, but a single easily identifiable reason is that they don’t solve the key/identity distribution problem.

2. The primary benefit of index-supplied hashing is for lockfiles. Specifically, they make lockfiles like uv.lock and pylock.toml useful/meaningful.

3. Is the source distribution problem, and is sort of fundamental to any packaging system that has any amount of dynamism in it. There’s a good argument to be made for reducing unnecessary dynamism (e.g. via wheels), but a nontrivial percent of Python users actually want this behavior.

Re: files: I think you’re getting confused about distribution files (sdists and wheels) versus the individual contents therein. You can’t modify distribution files on PyPI, you can only add new, unique ones. And this is now restricted to 14 days for each release.

[crabbone]

1. There was a discussion few years ago to remove the signatures from the format at all. I don't know what the resolution was, that's why I don't know if it's still supported, as in is it supported by the specification or not.

2. Lock files are hilariously wrong in Python. They don't accomplish what they claim because the only tool that in principle could do that and that is actually used with Python deployment / development is conda, and to the best of my knowledge conda's environment doesn't support locks. All other installers don't even guarantee that successfully running them will produce an environment with all dependencies met. But, not only that, they don't prevent users from becoming victims of manipulations that happen index-side because they don't store the checksums.

3. "nontrivial percent of Python users actually want this behavior." I'm yet to find one. Most Python users simply don't understand how any of their packaging works and come up with absurdly bad ideas and "solutions" to their problems. They may tell you they want something in the same sense as a drug addict may tell you they want more drugs, but the "actually" isn't there. They "actually" don't know what they want most of the time.

> You can’t modify distribution files on PyPI, you can only add new, unique ones.

I just gave an example of how this modifies the package. Not sure what problem do you see with it.

[edelbitter]

Would not count that as 20 years of sticking to that philosophy, though.

We also figured out 20 years ago that SHA1 was not quite as strong as initially estimated, and not quite 10 years ago that generating two colliding documents was merely a matter of some serious computing power. A few projects went ahead and changed the name of their master branch, but SHA256 preference remains elusive.

[winstonwinston]

Hash pinning (already) works, and this change is all about when you as a PyPI user do not use hash pinning for installing releases, when you pin just release version for example.

The release consists of one sdist and zero or more wheels. Until now you were able to upload additional wheels at later time.

[woodruffw]

Very pedantic of me, but I figure it’s interesting to note: technically a release on PyPI can have zero files or even one or more wheels but no sdist. The former is a degenerate case that users don’t normally see, and the latter happens if the user chooses to only upload wheels (or their sdist upload fails for whatever reason).

(This doesn’t change your observations at all! Just as a demonstration of how Python packaging’s data model can be unintuitive.)

[zbentley]

I'd argue that the latter's a degenerate case as well--for the somewhat nitpicky/minor reason that package managers' errors are extremely poor at indicating the difference between "the version you asked for doesn't exist" and "there are no artifacts compatible with your platform at the version you asked for (and, for bonus points, here are some artifacts that are compatible with your platform)".

Something that I wish was included in PEP-694 is the notion of a "tombstone" invalid/empty sdist artifact to deal with that case, to make it easier for package managers to interpret and surface those different failure classes.

Like, sure, this is a package manager UX problem, but it's been happening for long enough that I feel like it might be time for PyPI to help managers do better.

[tyingq]

BitTorrent used hash pinning earlier than that even.