[woodruffw]

> Where the only official workflow is "Use GitHub Actions".

The standard behind this (PEP 740) supports anything that can be used with Trusted Publishing[1]. That includes GitLab, Google Cloud, ActiveState, and can include any other OIDC IdP if people make a good case for including it.

It's not tied to Microsoft or GitHub in any particular way. The only reason it emphasizes GitHub Actions is because that's where the overwhelming majority of automatic publishing traffic comes from, and because it follows a similar enablement pattern as Trusted Publishing did (where we did GitHub first, followed by GitLab and other providers).

[1]: https://docs.pypi.org/trusted-publishers/

[belval]

I get that, that's why I didn't go "This is Embrace Extend Extinguish", but as constructive feedback I would recommend softening the language and to replace:

> STOP! You probably don't need this section;

In https://docs.pypi.org/attestations/producing-attestations/#t...

Perhaps also add a few of the providers you listed as well?

> The only reason it emphasizes GitHub Actions is because that's where the overwhelming majority of automatic publishing traffic comes from

GitHub being popular is a self-reinforcing process, if GitHub is your first class citizen for something as crucial as trusted publishing then projects on GitHub will see a higher adoption and become the de-facto "secure choice".

[HelloNurse]

PyPI should support and encourage open infrastructure.

If I don't want to use GitHub, let alone GitHub Actions, I am now effectively excluded from publishing my work on PyPI: quite unacceptable.

[woodruffw]

That’s now how any of this works. I am begging you to re-read the docs and understand that this does not require anybody to use GitHub, much less GitHub Actions, much less Trusted Publishing, much less attestations.

You can still, and will always be able to use API tokens.

[lucideer]

> I am begging you to re-read the docs

The gp was pointing out that the docs heavily emphasise (& therefore encourage) GHA usage & suggested language changes.

If people are confused about what they need to use Trusted Publishing & you're suggesting (begging) a re-read as the solution, that seems evidence enough that the gp is correct about the docs needing a reword.

[nmstoker]

It could just as easily imply that people aren't paying attention when they read it. Inability to understand a text is not always on the author, plenty of times it's on the reader.

[st3fan]

Well, yes and no. From the perspective of an infosec professional who is focussed on supply chain security I can tell you that your package having an attestation from a trusted platform like GitHub or GitLab gives me a warm feeling. It is not the only thing we will look at but definitely part of a longer list of checks to understand risk around dependencies.

With an attestation from GitHub I know at least that the workflow that ran it and the artifacts it produced will be 100% traceable and verifyable. This doesn't mean the code was not malicious, but for example it will rule out that someone did the build at home and attached an alternative version of an artifact to a GitHub release. Like how that was done with the xz project.

It is fine to not like GitHub, but I think that means we need more trusted builders. Developers cannot be pushed toward just GitHub.

[woodruffw]

> It is fine to not like GitHub, but I think that means we need more trusted builders. Developers cannot be pushed toward just GitHub.

Yes, agreed. This is why the docs explicitly say that we’re planning on enabling support for other publisher providers, like GitLab.

[ziddoap]

Thank you for being patient with people that seem to have willfully not read any of the docs or your clarifying comments here, are saying you are lying, and/or are making up hypothetical situations. It's appreciated!

Edit: woodruffw is sitting here and thoughtfully commenting and answering people despite how hostile some of the comments are (someone even said "This is probably deserving a criminal investigation"! and it has more upvotes than this comment). I think that should be appreciated, even if you don't like Python.

[HelloNurse]

I know attestations are not mandatory, but the rug has already been pulled: PEP 740 distinguishes "good" and "bad" packages and "good" packages require GitHub.

[cpburns2009]

Attestations are worthless unless they're checked. I have no doubt they'll eventually become the default in pip which effectively makes them mandatory for 99% of people not willing to jump through the hoops of installing an unattested package.

[woodruffw]

It does nothing of the sort, and the current GitHub requirement is an explicitly temporary restriction, like it was for Trusted Publishing. Again: I am begging you to read the docs that we’ve compiled for this.

[ziddoap]

This is exactly what I meant when I said "people that seem to have willfully not read any of the docs or your clarifying comments here"

[ljdgah]

Since roughly 2014 there have been so many rug pulls in the Python organization that no one believes anything any more.

It always starts small: "Oh, we just want everyone to be nice, so we have this super nice CoC document. Anyone who distrusts us is malicious."

Ten years later you have a corporate-backed inner circle that abuses the CoC to silence people, extend their power and earning potential and resort to defamation and libel.

It is possible that the people here who defend this corporate attestation-of-provenance-preferably-on-our-infrastructure scheme think that nothing nefarious will happen in the future. Well, given the repressive history of Python they are naive then.

It just takes pip to add a flag to enable "non-attested" packages. And of course they'll name it something like --allow-insecure-potentially-malicious.

[WhyNotHugo]

They’re not encouraging open infrastructure. In fact, this whole design doesn’t even contemplate the possibility of self hosting. Trust must be blindly delegated to one of the existing partners.

[woodruffw]

> but as constructive feedback I would recommend softening the language and to replace:

I can soften it, but I think you're reading it excessively negatively: that warning is there to make sure people don't try to do the fiddly, error-prone cryptographic bits if they don't need to. It's a numerical fact that most project owners don't need that section, since most are either using manual API tokens or are publishing via GitHub Actions.

> Perhaps also add a few of the providers you listed as well?

They'll be added when they're enabled. Like I said in the original comment, we're using a similar enablement pattern as happened with Trusted Publishing: GitHub was enabled first because it represents the majority of publishing traffic, followed by GitLab and the others.

> GitHub being popular is a self-reinforcing process, if GitHub is your first class citizen for something as crucial as trusted publishing then projects on GitHub will see a higher adoption and become the de-facto "secure choice".

I agree, but I don't think this is PyPI's problem to solve. From a security perspective, PyPI should prioritize the platforms where the traffic is.

(I'll note that GitLab has been supported by Trusted Publishing for a while now, and they could make the publishing workflow more of a first class citizen, the way it is on GHA.)

[belval]

> I agree, but I don't think this is PyPI's problem to solve. From a security perspective, PyPI should prioritize the platforms where the traffic is.

To me that's a bit of a weird statement, PyPI is part of the Python foundation, making sure that the project remains true to its open-source nature is reasonable?

My concern is that these type of things ultimately play out as "we are doing the right thing to limit supply chain attacks" which is good an defendable, but in ~5 years PyPI will have an announcement that they are sunsetting PyPI package upload in favor of the trusted provider system. pip (or other tooling) will add warnings whenever I install a package that is not "trusted". Maybe I am simply pessimistic.

That being said we can agree to disagree, I am not part of the PSF and I did preface my first comment with "I guess I am an idealist".

[woodruffw]

> making sure that the project remains true to its open-source nature is reasonable?

What about this, in your estimation, undermines the open-source nature of PyPI? Nothing about this is proprietary, and I can't think of any sane definition of OSS in which PyPI choosing to verify OIDC tokens from GitHub (among other IdPs!) meaningfully subverts PyPI's OSS committment.

> PyPI package upload in favor of the trusted provider system. pip (or other tooling) will add warnings whenever I install a package that is not "trusted". Maybe I am simply pessimistic.

Let me put it this way: if PyPI disables API tokens in favor of mandatory Trusted Publishing, I will eat my shoe on a livestream.

(I was the one of the engineers for both API tokens and Trusted Publishing on PyPI. They're complementary, and neither can replace the other.)

[mananaysiempre]

> What about this, in your estimation, undermines the open-source nature of PyPI?

Absence of support for self-hosting, in the spirit of freedom 0 = OSD 5&6? Or, for that matter, for any provider whose code is fully open source?

[woodruffw]

> Absence of support for self-hosting, or for that matter for any non-proprietary service?

This has nothing to do with self-hosting, whatsoever. You can upload to PyPI with an API token; that will always work and will not do anything related to Trusted Publishing, which exists entirely because it makes sense for large services.

PyPI isn't required to federate with the server in my basement through OpenID Connect to be considered open source.

[dogleash]

>> PyPI package upload in favor of the trusted provider system. pip (or other tooling) will add warnings whenever I install a package that is not "trusted". Maybe I am simply pessimistic.

> Let me put it this way: if PyPI disables API tokens in favor of mandatory Trusted Publishing, I will eat my shoe on a livestream.

Yeah, sure. But parent poster also mentioned pip changes. Will you commit to eating a shoe if pip verifies attestations? Of course not, you know better than I do that those changes to pip are in the works and experimental implementations already available. You must have forgotten to mention that. PyPI doesn't have to be the bad guy on making sure its a PITA to use those packages. Insert Futurama "technically correct, the best kind of correct" meme.

[woodruffw]

> You must have forgotten to mention that.

Insinuating dishonesty is rude, especially when it’s baseless: the post linked in this one is explicit about experimenting with verification. There’s no point in signing without verification.

pip may or may not verify attestations; I don’t control what they do. But even if they do, they will never be able to mandate them for every project: as I have repeatedly said, there is no plan (much less desire) to mandate attestation generation.

[BiteCode_dev]

I'm with @belval on this one, it's ok to prioritize github, but people that want the standard to implement an alternative should not feel like they are doing something that may not be supported.

It kinda feels like that right now.

[woodruffw]

Again, to be clear: the standard does not stipulate GitHub or any other specific identity providers. The plan is to enable GitLab and the other Trusted Publisher providers in short order.

This is exactly the same as Trusted Publishing, where people accused the feature of being a MSFT trojan horse because GitHub was enabled first. I think it would behoove everybody to assume the best intentions here and remember that the goal is to secure the most people by default.

[int_19h]

I think the point is that this needs to be made clearer in the official docs from the get go.

[woodruffw]

It's said explicitly in the second sentence in the usage docs[1].

> Attestations are currently only supported when uploading with Trusted Publishing, and currently only with GitHub-based Trusted Publishers. Support for other Trusted Publishers is planned. See #17001 for additional information.

[1]: https://docs.pypi.org/attestations/producing-attestations/

[TheRealPomax]

At this point it should be fairly obvious that if you have to defend the phrasing in multiple threads here on HN, get some folks to help rephrase the current document instead so you can comment with "we updated the text to make it clear this is a first pass and more options are getting added to the doc soon".

If you draw an ugly cat, and someone tells you it's ugly, it doesn't matter how much you insist that it isn't, and the same is true for docs. It doesn't matter what your intention was: if people keep falling over the same phrasing, just rephrase it. You're not your writing, it's just text to help support your product, and if that text is causing problems just change it (with the help of some reviewers, because it's clear you think this is phrased well enough, but you're not the target audience for this document, and the target audience is complaining).

[aseipp]

Anyone can run an OIDC system if they want. But PyPI is not under an obligation to trust an OIDC provider running on a random rpi3 in your basement. More than that, GitHub is "trusted" because we can be pretty sure they have an on-call staff to handle incidents, that they can reliably say "This token was provided on behalf of this user at this time for this build", etc.

Even if you standardized the more technical parts like OIDC claim metadata (which is 100% provider specific), it wouldn't really change the thrust of any of this — PyPI is literally trusting the publisher in a social sense, not in some "They comply with RFC standards and therefore I can plug in my random favorite thing" sense.

This whole discussion is basically a non-issue, IMO. If you want to publish stuff from your super-duper-secret underground airgapped base buried a mile underneath the Himalayas, you can use an API token like you have been able to. It will be far less hassle than running your own OIDC solution for this stuff.

[immibis]

If I can't build on a rpi3 in my basement and am forced to use GitHub that's exactly against the spirit of open source

[colejohnson66]

You still can. You just use an API token with PyPI.

[aseipp]

Please improve your reading comprehension. I swear, this website is embarassing sometimes. You can still do this with an API Token. You can upload from a C64 with an API token. What you cannot do is run some random OIDC provider on your random useless domain and have PyPI magically respect it and include as part of the Trusted Publishers program. There is no point in it, because the program itself is constrained by design because it only provides any benefit at "large scale." Your random dumb server providing a login for you alone does not provide any benefits over you just using an API Token.

Any pathway to provide trusted attestations for random individual Hacker News users like yourself will, in fact, require a different design.

[guappa]

> error-prone cryptographic bits if they don't need to

They can't. Because you wouldn't accept their key anyway.

[woodruffw]

I think you're missing something. The key in question is a short-lived ECDSA key that lives inside a publishing workflow and is destroyed after signing; neither GitHub nor the Sigstore CA generates a signing key for you.

PyPI will accept any key bound to an identity, provided we know how to verify that identity. Right now that means we accept Trusted Publishing identities, and GitHub identities in particular, since that's where the overwhelming majority of Python package publishing traffic comes from. Like what happened Trusted Publishing, this will be expanded to other identities (like GitLab repositories) as we roll it out.

[guappa]

How does pypi know I'm not github? Because I can sign with my keys and not with github's key.

Never mind all the low level details of the temporary keys and hashes and all of that. This is an high level comment not a university book about security.

[freeone3000]

It requires an OIDC IdP, though… with PGP, I can verify that my identity is constant, but now, I’m reliant on some chosen third-party. And it has to keep being the same one, to boot! I don’t like the lock-in this causes, and I definitely don’t like the possibility of my right to publish being revoked by a third party.

[beng-nl]

I’m only now learning about what OIDC IdP is (for those like me: openid connect identity provider). But from my reading, a self hosted gitlab can function as an oidc idp.

That would be enough control, right?

[chippiewill]

You can't use a self-hosted Gitlab because you can only use a "trusted publisher".

There's no hard technical reason for that. It's mostly that PyPI only want to trust certain issuers who they think will look after their signing keys responsibly.

[intelVISA]

Sounds like PyPI have been corrupted? Hopefully The Foundation can remind them of their mission.

[guappa]

Good luck with that…

[woodruffw]

There is a technical reason for it, and it’s explained in an adjacent thread. Accepting every single small-scale IdP would result in a strictly worse security posture for PyPI as a whole, with no actual benefit to small instances (who are better off provisioning API tokens the normal way instead of using Trusted Publishing).

[chippiewill]

I don't think that's a technical reason, that's more of a social reason about trust. Adding configurable JWT issuers is not technically hard or difficult to do in a way that doesn't compromise PyPI's inherent security. There's a configuration difficulty, but those using it will generally know what they're doing and it only compromises their own packages (which they can already compromise by losing tokens etc.)

I disagree with your assessment that provisioning API tokens is better than being able to authenticate with a JWT. It makes managing credentials in an organisation much easier as far fewer people need access to the signing key compared to who would need access to the API token. Using asymmetric keys also means there's less opportunity for keys to be leaked.

[woodruffw]

Adding another IdP is not the hard part; establishing the set of claims that IdP should be presenting to interoperate with PyPI is. The technical/social distinction is specious in this context: the technical aspects are hard because the social aspects are hard, and vice versa.

If you work in a large organization that has the ability to maintain a PKI for OIDC, you should open an issue on PyPI discussing its possible inclusion as a supported provider. The docs[1] explain the process and requirements.

[1]: https://docs.pypi.org/trusted-publishers/internals/

[Spivak]

> who are better off provisioning API tokens the normal way

As long as those packages get digital attestation, perhaps attested by PyPI itself post-upload or from a well-known user provided key similar to how GPG worked but managed by PyPI this time.

Surely you see how this is creating two classes of packages, where the favored one requires you use a blessed 3rd party?

[woodruffw]

No, I don't. There's no plan, and there will be no plan, to make installers reject packages that don't have attestations. Doing so is technically and socially impossible, and undesirable to boot.

The strongest possible version of this is that projects that do provide attestations will be checked by installers for changes in identity (or regression in attestations). In other words, this feature will only affect packages that opt to provide attestations. And even that is uncertain, since Python packaging is devolved and neither I (nor anybody else involved in this effort) has literally any ability to force installers to do anything.

[zahlman]

"Ordinary user" here.

It comes across like the goal of this system is to prove to my users a) who I am; b) that I am working in cooperation with some legitimate big business. I don't understand why I should want to prove such things, nor why it's desirable for b) to even be true, nor why my users should particularly care about a) at all, whether I can prove it or not.

I think my users should only care that the actual contents of the sdist match, and the built contents of the wheel correspond to, the source available at the location described in the README.

Yes, byte-for-byte reproducible builds would be a problem, for those who don't develop pure Python. Part of why sdists exist is so that people who can't trust the wheel, like distro maintainers, can build things themselves. And really - why would a distro maintainer take I take "I can cryptographically prove the source came from zahlman, and trust me, this wheel corresponds to the source" from $big_company any more seriously than "trust me, this wheel corresponds to the source" directly from me?

If I have to route my code through one of these companies ("Don't worry, you can work with Google instead!" is not a good response to people who don't want to work with Microsoft) to gain a compliance checkmark, and figure out how someone else's CI system works (for many years it was perfectly possible for me to write thousands of lines of code and never even have an awareness that there is such a thing as CI) so that I can actually use the publishing system, and learn what things like "OIDC IdP" are so that I can talk to the people in charge...

... then I might as well learn the internal details of how it works.

And, in fact, if those people are suggesting that I shouldn't worry about such details, I count that as a red flag.

I thought open source was about transparency.

[solatic]

There's two alternate visions: anybody can work with anybody they trust, and Only BigCo Can Protect You.

We know the first falls apart because the Web of Trust fell apart. When the UI asks people "hey, this is new, here's the fingerprint, do you trust it?", nobody actually conducts review, they just mindlessly click "yes" and move on. Very, very, very few people will actually conduct full review of all changes. Even massive organizations that want to lie to themselves that they review the changes, end up just outsourcing the responsibility to someone who is willing to legally accept the risk then they go ahead and just mindlessly click "yes" anyway. There are just too many changes to keep track of and too many ways to obscure malicious code.

We know the second falls apart because human beings are both the inevitable and ultimate weakest link in even the robustest security architectures. Human beings will be compromised and the blast radius on these highly centralized architectures will be immense.

Pick your poison.

[woodruffw]

> And, in fact, if those people are suggesting that I shouldn't worry about such details, I count that as a red flag.

Nobody is suggesting that. There's an annotation on the internals documentation to make sure that people land on the happy path when it makes sense for them. If you want to learn about the internals of how this works, I heartily encourage you to. I'd even be happy to discuss it over a call, or chat, or any other medium you'd like: none of this is intended to be cloak-and-dagger.

(You can reach this conclusion abductively: if any of this was intended to be opaque, why would we write a public standard, or public documentation, or a bunch of high-profile public announcements for it?)

[dogleash]

> Part of why sdists exist is so that people who can't trust the wheel, like distro maintainers, can build things themselves.

Some might even expect pypi to build everything going into it's repo similar to a distro building for it's repos. As far as I can tell, all this 'attestation' is just pypi making microsoft pay for the compute time with extra steps.

[guappa]

Except that also for trusted publishing, they only allowed github in the beginning and eventually added a couple of other providers. But if you're not google or microsoft you won't be added.

[woodruffw]

These kinds of comments are borderline mendacious: you can observe, trivially, that 50% of the Trusted Publishers currently known to PyPI are neither Google nor Microsoft controlled[1].

If PyPI accepts two more likely ones, a full 2/3rds will unrelated to GitHub.

[1]: https://docs.pypi.org/trusted-publishers/adding-a-publisher/

[guappa]

Ping me when one of them will be an open source entity rather than a company.

[woodruffw]

https://docs.pypi.org/trusted-publishers/internals/#how-do-i...

[immibis]

Wow. I get to choose one from a total of FOUR large corporations! Amazing openness!

[woodruffw]

Once again: this is constrained by design. If you don’t want to use OpenID Connect, just create a token on PyPI and publish the normal way. You are not, and will never be, required to use this feature.

[cpburns2009]

Wow, you can use a whole two other providers from your list: Gitlab and ActiveState. Color me unimpressed.

[silverwind]

Why does this need to allowlist CI providers in first place? Why not publish an open interface any CI provider can integrate against?

[woodruffw]

Because the security benefit of Trusted Publishing via OIDC versus normal API tokens is marginal at small scales, in two senses:

1. The primary benefit of Trusted Publishing over a manual API token is knowing that the underlying OIDC IdP has an on-call staff, proper key management and rotation policies, etc. These can be guaranteed for GitHub, GitLab, etc., but they're harder to prove for one-off self-hosted CI setups. For the latter case, the user is no better off than they would be with a manual API token, which is still (and will always be) supported.

2. If the overwhelming majority of traffic comes from a single CI/CD provider, adding more code to support generic OIDC IdPs increases PyPI's attack surface for only marginal user benefit.

There also is no "open interface" for PyPI to really use here: this is all built on OIDC, but each OIDC provider needs to have its unique claims mapped to something intelligible by PyPI. That step requires thoughtful, manual, per-IdP consideration to avoid security issues.

[chippiewill]

I still think this is overly strict. Supporting arbitrary OIDC providers is not excessively complex or particularly rare, the major cloud providers all support it in one way or another [1][2][3], as does Hashicorp Vault [4]. I disagree that the primary benefit over a manual API token is _knowing_ that the OIDC IdP is following the best practices you talk about. Having it rely on asymmetric keys makes the process more secure and scalable than API tokens for those that choose to use it.

I think there's a separate question around trust. But I think blocking non-trusted publishers from using a more secure form of authentication isn't the answer. Instead I think it makes more sense to use nudges in the PyPI UI and eventually of consumers (e.g. pip) to indicate that packages have come from non-trusted publishers.

[1] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_pr... [2] https://learn.microsoft.com/en-us/graph/api/resources/federa... [3] https://cloud.google.com/identity-platform/docs/web/oidc [4] https://developer.hashicorp.com/vault/docs/auth/jwt

[guappa]

I can create a github account. How does that make me trustworthy? How being able to create a github account prevents me from adding a virus in my module?

[chippiewill]

It's not about the package maintainer, it's about the trustworthiness of the OIDC issuer to prove the identity of a user.

A poorly maintained issuer could leak their secret keys, allowing anyone to impersonate any package from their service.

[guappa]

But what use does it serve to prove that I am user "qioaisjqowihjdoaih" on github?

I mean it only proves I authenticated successfully. Nothing else.

[freeone3000]

I think I would be better off with API key + PGP than API key alone. And that’s being phased out?

[woodruffw]

You can no longer upload a PGP signature to PyPI, if that's what you mean. That was phased out last year (to virtually no complaint since nobody was actually verifying any of the signatures, much less attempting to confirm that their keys were discoverable[1]).

[1]: https://blog.yossarian.net/2023/05/21/PGP-signatures-on-PyPI...

[guappa]

> to virtually no complaint since nobody was actually verifying any of the signatures

And this is in no way a consequence of pypi stopping to host public keys right? Say the whole story at least… Say that there used to be a way to verify the signatures but you dropped it years ago and since then the signatures have been useless.

[woodruffw]

If it did, it was well before I ever began to work on PyPI. By the time I came around, PGP signature support was vestigial twice over and the public key discovery network on the Internet was rapidly imploding.

(But also: having PyPI be the keyserver defeats the point, since PyPI could then trivially replace my package's key. If that's the "whole story," it's not a very good one.)

[antononcube]

> [...] the user is no better off than they would be with a manual API token, which is still (and will always be) supported.

This is good to know. I did not see related statements in of the documents linked to this discussion, though.

[antononcube]

I am not sure why my comment above is downvoted -- if you know where the perpetual optionality of digital attestations is officially stated, please, provide a link.

[SethMLarson]

Because every CI/ID provider has a different set of claims and behaviors that would constitute a "secure" policy for verification. If there was one singular way to do that then we could, but there isn't yet so PyPI needs to onboard providers piecemeal. The work to add a new provider is not massive, the reason there are not tons of providers isn't because the work is hard but rather because people are voting with their feet so Github and Gitlab make sense as initial providers to support.

[benatkin]

That's good. IIRC the feature was launched with GitHub as the only option, which is the same state of affairs of https://jsr.io/ right now.

IMO the launch should have been delayed until there was more than one trusted publisher.