[lrvick]

Had the Nix team rolled out signed commits/reviews and independent signed reproducible builds as my (rejected) RFC proposed, then it would not be possible to do any last mile supply chain attacks like this.

In the end NixPkgs wants to be wikipedia easy for any rando to modify, and fear any attempt at security will make volunteers run screaming, because they are primarily focused on being a hobby distro.

That's just fine, but people need to know this, and stop using and promoting Nix in security critical applications.

An OS that will protect anything of value must have strict two party hardware signing requirements on all changes and not place trust in any single computer or person with a decentralized trust model.

Shameless plug, that is why we built Stagex. https://stagex.tools https://codeberg.org/stagex/stagex/ (Don't worry, not selling anything, it is and will always be 100% free to the public)

[pyrox]

Hey! First, a disclaimer: I do not speak for anyone officially, but I am a very regular contributor to nixpkgs and have been involved in trying to increase nixpkgs' security through adopting the Full-Source Bootstrap that Guix and Stagex use. I also assume that the RFC you're talking about is RFC 0100, "Sign Commits"(ref: https://github.com/NixOS/rfcs/pull/100)

As mentioned in the RFC discussion, the major blocker with this is the lack of an ability for contributors to sign from mobile devices. Currently, building tooling for mobile devices is way out-of-scope for nixpkgs, and would be a large time sink for very little gain over what we have now. Further, while I sign my commits because I believe it is a good way to slightly increase the provenance of my commits, there is nothing preventing me from pushing an unsigned commit, or a commit with an untrusted key, and that's, in my opinion, fine. While for a project like Stagex(which as a casual cybersecurity enthusiast and researcher, I thoroughly appreciate the security work you all do), this layer of security is important, as it's clearly part of the security posture of the project, nixpkgs takes a different view to trustworthiness. While I disagree with your conclusion that having this sort of security measure would "make volunteers run screaming", I would be interested in seeing statistics on the usage of these mechanisms in nixpkgs already. Nixpkgs is also definitely not focused on being a hobby distro, considering it's in use at many major companies around the world(just look at NixCon 2025's sponsor list).

To be clear, this isn't to say that all security measures are worthless. Enabling more usage of security features is a good thing, and it's something I know folks are looking into(but I'm not going to speak for them), so this may change in the future. However, I do agree with the consensus that for nixpkgs, enabling commit signing would be very bad overall for the ecosystem, despite the advantages of them. Also, I didn't see anything in your PR about "independent signed reproducible builds", but for a project the size of nixpkgs, this would also be a massive infrastructure undertaking for a 3rd-party, though NixOS is very close to being fully reproducible(https://reproducible.nixos.org/) at the moment, we're not there yet though.

In conclusion, while I agree that signing commits would a good improvement, the downsides for nixpkgs are significant enough that I don't believe it would be a good move. It's something to definitely continue thinking about as nixpkgs and nix continue to refine and work on their security practices, though. I would also love some more information about how Stagex does two-party hardware signing, as that sounds like something interesting as well. Thank you so much!

Edit: Also, want to be very clear: I am not saying you're entirely wrong, or trying to disparage the very interesting and productive work that Stagex is doing. However, there were some (what I felt were)misconceptions I wanted to clean up.

[vlovich123]

The reason I dislike this is this is the first thing in the article:

> in nixpkgs that would have allowed us to pwn pretty much the entire nix ecosystem and inject malicious code into nixpkg

OP provided a mechanism to stymie the attack. The counter from your position needs to be how the nix project otherwise solves this problem, not “this isn’t the right approach for hand wavy reasons”. Given the reasonings stated, OP has convinced me that Nix isn’t actually serious about security as this should be treated as an absolutely critical vulnerability that has several hardening layers wrapped to prevent such techniques.

[tennysont]

> in nixpkgs that would have allowed us to pwn pretty much the entire nix ecosystem and inject malicious code into nixpkg

Isn't that what happens when a build server or source code is compromised? I'm not sure if the existence of this exploit was egregious, but the blast radius seems normal for a build server exploit.

> how the nix project otherwise solves this problem

You can go into `/etc/nix/nix.conf` and remove `trusted-public-keys` so that you don't trust the output of the build servers. Then you just need to audit a particular commit from nixpkgs (and the source code of the packages that you build) and pin your config to that specific commit.

Otherwise, it seems like the solution is to harden the build system and source code control so that you can freely trust the source code without auditing it yourself. I'm not sure what else can be done.

If your threat model is that the 10+ nixpkg contributors are trustworthy but the github repo is untrustworthy, then git signing would make you safe.

Personally, I worry that a carelessly approved merge in nixpkg or an upstream supply attack is a bigger threat then a github repo exploit (as described here), but I imagine that reasonable minds could disagree.

Regardless, I'm very excited to see that nix builds are almost fully reproducible. That seems great! It seems like this could potentially be the foundation on which a very secure distro is built.

[lrvick]

You absolutely should never trust a centralized build server. Any security critical software distribution process should have all packages independently built, verified to have identical hashes, and signed by systems controlled by as many different trusted maintainers or third parties as possible.

Then any user can prove the binary they got was built faithfully from source due to those redundant build system signatures. We designed ReprOS for this purpose.

stagex has also been 100% deterministic, full source bootstrapped, and independently reproduced/signed by multiple maintainers since our first release with a small team of 10ish regular contributors, so it can be done.

[typpilol]

Their leadership and community is also a disaster

[rendaw]

> the major blocker with this is the lack of an ability for contributors to sign from mobile devices

Do you mean a significant number of nixpkgs contributors make nixpkgs PRs from their phones... via the github web editor?

That seems weird to me at face value... editing code is hard enough on a phone, but this is also for a linux distro (definitely not a mobile os today), not a web app or something else you could even preview on your phone.

Edit: Per https://docs.github.com/en/authentication/managing-commit-si... the web editor can/does sign commits...

[Xylakant]

Mobile devices are not restricted to phones, but include tablets, some of which are pretty powerful and usable for code editing.

Note that the signature for the web interface is made with a GitHub owned key on your behalf and not with your personal key.

[lrvick]

Sorry to be that guy, but if someone cannot afford a $10 bit of hardware for the most basic attempt at protecting others from being harmed by someone impersonating them... then they have no business being a trusted maintainer in a Linux distribution relied on for billions of dollars in infrastructure.

That would be like someone saying they could not afford a mask in COVID or something. It is hard to believe these people really exist. I could go find $10 in change looking on the ground of a few nearby fast food pick-up windows, because I have done it. Many times. Free money!

Anyway, such people will be easy to bribe, easy to target, easy to steal from. Letting that sort of person have trust in a major OS is endangering them, and frankly irresponsible.

For anyone that makes excuses about being unable to produce a hardware signing device, of course let them contribute, but then let two confirmed real humans with hardware keys adopt, review, and sign that PR, and always have at least two real confirmed humans with hardware keys sign every change both as code, and as reproducible artifacts after.

We have taken in tons of drive-by unsigned contributions in stagex. This is no problem. We just pretend an AI bot wrote it, and require one maintainer to "adopt" the commit to sign it (maintaining attribution), and then a second maintainer reviews, and does a signed merge as usual.

[lrvick]

My RFC was much earlier in 2018 https://github.com/NixOS/rfcs/pull/34

Lack of supply chain integrity controls as a means to reduce contribution friction to maximize the number of packages contributed is a perfectly valid strategy for a workstation distribution targeted at hobby developers.

Volunteers can do what they want, so that RFC convinced me stagex needed to exist for high security use cases, as Nix was explicitly not interested in those.

This is all fine. The reason I speak in a tone of frustration whenever Nix comes up is because as a security auditor I regularly see Nix used to protect billions of dollars in value or human lives. Sysadmins uneducated on supply chain integrity just assume Nix does security basics and has some sort of web of trust solution like even OG distros like Debian, but that is just not the case.

Nix maintainers did not ask to be responsible for human lives and billions in value, but they are, and people will target them over it. I am afraid this is going to get people hurt.

https://github.com/jlopp/physical-bitcoin-attacks

Nix choosing low supply chain security to maximize the total number of packages endangers themselves and others every time someone ignorantly deploys nix for high value applications.

If nix chooses to maintain their status quo of no commit signing, no review signing, no developer key pinning, and no independent reproducible build signing, they need to LOUDLY warn people seeking to build high risk systems about these choices.

Even those basic supply chain controls which we use in stagex, are nowhere near enough, but they are the bare minimum for any distro seeking to be used in production.

[tennysont]

Out of curiosity, why don't/didn't you start a new version of nixpkgs with hardened source? You could forgo the build server, forcing users to build from scratch (at least to start). You could leverage the plentiful, albeit, less secure, packaging code in the nixpkgs to quickly build out your hardened versions.

Effectively, you're building out an audited copy of nixpkgs on the same build engine, but with hardened configs. Write wrappers to validate git signatures when users update, and you got yourself a chain of trust on the source code distribution for your hardened nixpkg.

I'm sure you had reasons, I'm just interested to know your thought process.

[lrvick]

I ultimately thought out what would be easier, a decade political fight to make massive changes to nix, or a fork of it written solo to improve auditability and security, or starting over from the top with a design that checks every dream box I wanted from a linux distro.

I had many RFCs that would have followed this rejected one if there was any change tolerance... so my fastest path to prove out my ideas for a distro with decentralized trust was to start one with that explicit goal.

If I wanted to make things maximally auditable and portable to different build engines, a published dead simple spec with multiple competing implementations that most software engineers already know how to write would be ideal. People could review an engine they use, or ensure all existing implementations on any operating system get identical results and are thus trusted that way. If it natively supports a ton of features to make deterministic builds wildly simpler, major bonus.

OCI/Containerfile was a check on all fronts, and some early maintainers and I riffed on design patterns and realized the OCI ecosystem already had specified multi party signing and verification, artifact integrity, smart layer by layer caching etc etc. This fit our dev experience and threat model perfectly and we could just skip implementing the package build and distribution layer and just start writing packages, like that day. None of us needed to learn or invent a new language or ask auditors to do so or fork nix ecosystem to have proper signing support and write a sane spec... that could be years of wheel spinning.

The time saved by choosing an existing widely used and implemented spec meant we were able to put all energy into full source bootstrapping, universal multi party hardware signing on every build, change, review, and reproduction. Just full source bootstrapped linux from scratch in containerfiles with OCI native multi party signing if all parties independently get the same oci hashes from local builds. Oh and we are going LLVM native like Chimera next week. Big sweeping changes like that are easy with our ultralight setup.

I would note that the features we need for deterministic builds in docker, the most popular OCI implementation, only landed a couple of months before we started stagex, and the full source bootstrapping work by the bootstrappable builds team only got a complete bootstrap for the first time a few months before that and Guix shortly after. Tons of reference material.

If stagex had started before 2022 I imagine we might have used a heavily trimmed down nix clone or tried to convince guix to adopt our threat model, which is much further along in supply chain security than nix but scheme would have been a very isolating choice. I think stagex got lucky starting at exactly the right time when two huge pieces of the puzzle were done for us.

[otabdeveloper4]

NixOS is miles better from a security standpoint than any Debian or Red Hat already, so take what you can.

[lrvick]

It is way behind Debian on even the basics sadly. Maintainers do not even sign in NixOS making them easy to impersonate. Debian security is a joke too though in other areas, and like nix, should never be used in production either.

See a security comparison of both with stagex: https://codeberg.org/stagex/stagex#comparison

[otabdeveloper4]

> should never be used in production either

A very hot and very wrong take.

NixOS at least has immutable read-only system images. This makes it a thousand times less interesting to a potential attacker than a Debian system.

For every Mossad agent crafting elaborate impersonation scheme to steal state secrets, there are a million script kiddies looking for insecure servers for a botnet.

P.S. A bigger issue is the complete inability of the "security industry" to understand even basic threat model issues. More proof that this entire "industry" is a joke and a clown show.

[XorNot]

Wow...this is possibly exactly what I've wanted to do for a while, but you already did it!

[gmfawcett]

That's pretty impressive -- thanks for sharing the link.

[cpuguy83]

Just a word of encouragement here, this is super interesting!

[friendly_wizard]

Stagex is honestly the most interesting discovery in this thread