[PunchyHamster]

There was no bug or attack on Debian since 2007 that reproducible packages would prevent.

"Well worth it" is not correct. And it just ups the the contribution barrier to Debian higher, I already heard a lot of people complaining that contributing to Debian is hard and while in past I defended it by "they need all the checks and bounds to make sure packages play with eachother nicely", this is just step that makes it hard for no reason and little benefit.

[savolai]

” If you are wondering why we are doing this at all, then hopefully the Reproducible Builds website will explain why this is useful.”

https://reproducible-builds.org/

Could you perhaps respond to the argumentation here?

[dvogel]

(Not OP, but...) I still fail to see the current value in confirming that a reproducing builder also included the same compromised dependency that I did when I built it. I understand that reproducible builds are guarding against dynamic attacks within build infrastructure. However I just don't see those happening. Compromised source dependencies are a 100x more common problem.

[ckastner]

I agree that compromised source dependencies are the bigger problem, but that doesn't mean a compromised build infrastructure isn't. Just this last week, we had two Linux kernel LPEs that could have been leveraged to implement just such an attack, for example.

Another thing to consider is that Debian has quite a few derivatives who may also rebuild packages from source, so you have a multiplier there.

[fsflover]

https://en.wikipedia.org/wiki/XZ_Utils_backdoor

[PunchyHamster]

that's example of attack reproducible packages do not protect from, why you are linking it ?

[fsflover]

A distro automatically verifying that installed packages are reproducible would protect the user?

[dvogel]

No, it wouldn't. The xzutils attacker compromised the source repository. The build pipeline portions were used to obscure the purpose of the exploit embedded in the source code repository.

[theteapot]

In xz-utils hack the attacker slipped changes into the Github release tarball that were not present in the Github version / git commit history. The Debian maintainer built from the release tarball instead of just pulling from the git repo directly. Shouldn't have been doing that but good luck convincing him not to use the workflow he's been using for the last X years (I tried). With repro builds we can clone the git directly confirm we get the same build.

[mmooss]

https://news.ycombinator.com/item?id=48083768

[PunchyHamster]

I know why they are useful. I am arguing they are waste of time for effort involved.

Forcing devs to use hardware keys to sign commits/CI requests would be actual security improvement, thwarting many supply chain attacks that only worked coz the attacker got to developer credentials. Hardware keys at least have option to make some operations require physically pressing the key so there is chance developer will notice.

But thanks to reproducible builds, at least someone can... validate that the binary code of vulnerable package can be reproduced. Very fucking useful.

I am not saying it is useless. I am saying it is one of highest hanging fruits on security tree.

[uecker]

A hardware key does not help if the developer's machine is compromised, as there is no change to understand what is signed anymore, or do you think the hardware key will show all the source code on its little display before signing?

With reproducible builds, you do not need to trust that the system that build the binary was not compromised, because this would be detected immediately.

Source compromises are still an issue, but there is a much bigger change that they are detected. Also if there is a compromise, reproducible builds allow you to later track it to the source. For an infected binary it is much more difficult to understand how it got there and what else might be compromised.

[PunchyHamster]

The way at least YK works is that you can set it up that pressing a key is requires for signing so at least "silently steals creds and sends malicious code" case (which is vast majority of compromises) gets handled

> Also if there is a compromise, reproducible builds allow you to later track it to the source.

They do not. Git log and build logs allow for that.

Reproductive builds only have value after the source. They protect build servers from being compromised (and then only if some other uncompromised environment is also running verification passes), if the bug is at source reproductive builds are exactly as valuable as writing commit that was used for build in app's code/package metadata.

[uecker]

If your compiler (or other tool or automatic build environment) is compromised and inserts a backdoor in the binary during building, the fact that you need to hold a key while signing or not is completely irrelevant.

git log and build logs do not help you at all, if you can not even determine that the compromised binary has any relation to the build log or the source you may want to look it. This is what reproducible builds give you. You are right that it does not protect against compromised sources.

[rstuart4133]

> I know why they are useful. I am arguing they are waste of time for effort involved.

Not being reproducible is a bug.

There is no reason for a build to not be reproducible, but somehow we let the built binaries become infested with timestamps, login names and file system paths. We recently moved to reproducible builds at work, and discovered that our login names and local home directory paths were being shipped in every release. No one was was very happy about leaking PPI like that.

You may not consider it worth the effort, but you aren't the one putting in the effort so I'm not sure why that matters to you. It is very much worth the effort to those people doing the work. Debian is a do-orcracy and so the people doing the work get to make the decisions.

[pseudohadamard]

Sure. The site appears to be a bunch of warm-fuzzies that could apply to almost any other measure you take, it's nothing specific to reproducible builds. As the original poster said "There was no bug or attack on Debian since 2007 that reproducible packages would prevent". In fact, it could be argued that reproducible builds lead to a reduction in security, not an improvement: They give an attacker an exact fixed memory layout for all of the binaries, so if you develop something like a ROP exploit for a copy on your local system you know that exploit will work on every other system as well because the binary layout is identical. It's a perfect monoculture where everything is vulnerable to the same exploit. It seems to have been something created by geeks to impress other geeks, without any considerations for whether it has any actual benefit or not.

[stabbles]

This comment is misinformed. Non-deterministic builds would also result in one tarball redistributed to all distro users. The ROP exploits don't work because of ASLR.

[pseudohadamard]

ASLR makes ROP attacks harder, it doesn't stop them, as a great many successful attacks have demonstrated. Heck, bypassing ASLR is taught to students at MIT... can't find the direct link ATM but here's a student assignment, https://csg.csail.mit.edu/6.S983/labs/aslr/.

[uecker]

This does not make your comment above less wrong.

[MomsAVoxell]

Reproducible builds are applicable not only to respond to ‘attacks’, a subject you seem to be bikeshedding, but also for other reasons too.

Anyone having to maintain a code base or a distributed fleet of devices will gain from this decision, immensely, as their operational periods come and go.

Reproducible builds are about longevity as much as they are about security.

Please don’t make bold claims about ‘no reason and little benefit’ while demonstrating ignorance of this hard fact: reproducible builds should have been the norm, in computing, from the get-go.

[bluGill]

I longevity is harmed though. Your certs need to expire in a few years we think that your toolchain will not be downloadable.

[yjftsjthsd-h]

> your toolchain will not be downloadable.

Why not? Debian has a fantastic track record of providing old versions, for instance here's the build tools from Debian 2.0 from 1998:

https://mirrors.accretive-networks.net/debian-archive/debian...

[MomsAVoxell]

Those problems need to be solved as well.

[bluGill]

I don't think they do, actually. Longevity sounds good, but in reality anything that's old probably has critical security holes and so you shouldn't use it anyway.

[iamnothere]

A warning is sufficient. Old tech should continue to work, for preservation and archival reasons.

[tardedmeme]

I've long ago realized that archival needs to be a separate task left to archivists and archive systems. If you take it into account when designing a live system it's liable to seriously compromise your system design.

Say you're making a chat app - you wouldn't incorporate a delete feature, and you might be tempted to use some kind of blockchain to prove all messages were delivered without gaps. But if you ignore archival needs you design something similar to IRC which is much simpler.

[PunchyHamster]

just archiving binary artifacts and source packages is enough

Reproducibility adds nothing here

[MomsAVoxell]

You probably don’t know how industrial computing works.

Toolchains absolutely need to be maintained with some degree of longevity.

The whole world doesn’t march to your consumer-user beat. Sometimes it functions at industrial-user tempo’s, too ..

[MomsAVoxell]

You’re not thinking like an industrial user but rather as a consumer. Maybe you should extend your scope a little bit.

[bluGill]

Industry is learning - often the hard way that out of date software is only acceptable if the device is not connected to a network at all. Even government labs with a separate top secret network that isn't supposed to be connected to anything else get hacked from the internet.

Not that you are wrong, industry keeps thinking they can make themselves immune and so long term reproducibility is useful, but I submit they are wrong.

[PunchyHamster]

> Anyone having to maintain a code base or a distributed fleet of devices will gain from this decision, immensely, as their operational periods come and go.

Just baking in build ID and commit is enough. What you think reproducible builds add ?

> Please don’t make bold claims about ‘no reason and little benefit’ while demonstrating ignorance of this hard fact: reproducible builds should have been the norm, in computing, from the get-go.

So far not a single person in the thread gave me concrete example (as in existing project, existing problem, no other solution can solve it). Just claiming it's better based on their feelings. Come on, be the first one.

[MomsAVoxell]

I already gave you an example, you dismissed it because you know better, but it is clear that you haven’t thought this through from the perspective of systems designers who have to deploy a base OS, with expected lifetimes of years, across a large fleet of devices.

Think industrial applications, such as rail and heavy industry transportation. We use reproducible builds here as part of a wider safety-critical protocol which guarantees that what we are running is what we expect to run - nothing more, nothing less.

Reproducible builds are certifiable. They can be relied on in environments where certification costs millions and takes years.

Think outside your consumer box for a minute.

[azkalam]

Reproducible builds reduce the need for trusted parties.

Have many organizations produce the binaries independently and post the arifacts.

Once n of m parties agree on the arifact hash, take that as the trusted build.

If every party reaches a different hash then we cannot build consensus.

[sgc]

To move away from organizational dependence, there should be an installable project for debian where I can dedicate some configurable small percentage of my compute when idle to reproducibly building debian components to make a robust verification system, starting with the most critical code.

Obviously, it would be a ton of work to make such a system resistant to gaming by malicious actors (see GNU Guix for useful efforts), but it would provide valuable diversity in architecture and (political or other) control.

It would be even cooler if we could have independent projects that could run on various distros and OS, and build packages for any of them. Having packages for bsd verified on linux and vice-versa with statistical logging (this code has been verified x times on y OSes) would be reassuring.

[PunchyHamster]

I think that project is called Ubuntu.

[sgc]

I don't know of anything Ubuntu is doing that is significantly beyond what Debian is doing in this regard, nor that they have a distributed reproduction system set up???

[saghm]

Building Ubuntu does not produce identical binaries to Debian, so no, that's not what they're asking for

[benregenspan]

Is the "Jia Tan" XZ Utils compromise not a good example? That relied on code snuck into a release that was not in source.

(It was caught before being promoted into a stable Debian release, yes, but this sort of relied on a happy accident, too close for comfort)

[chuckadams]

The xz hack was still reproducible, because it was included in the distribution archive which did not match the upstream source -- even then, it was so obfuscated it likely would have gone unnoticed, but nevertheless it only lived in the uploaded tarball and not in the repo. Reproducibility is a good thing, but the next step is build provenance.

Still, lots of good non-security benefits to reproducible builds too.

[jcranmer]

The xz utils compromise is a very good example... of why reproducible packages doesn't actually solve anything security-wise!

The backdoor relied first on a difference between building a package in a packaging environment versus building the package on your own. And also, it relied on the very common practice of checking in unreviewable artifacts into the source tree (e.g., the configure script, malicious binary test artifacts).

Reproducible builds guarantee that two people can follow the same instructions and get the same, bit-identical outcome. It does nothing to guarantee that those instructions have not been compromised, and all of the great packaging security failures of my lifetime that I can think of have relied on those instructions being compromised (e.g., xz utils, Debian OpenSSL keygen issues).

[uecker]

An attack would be far easier without reproducible packages. One could upload a compromised binary to debian by becoming a debian developer, blackmail a debian developer to so, or compromise the computer of a debian developer or the distribution.

At the time of xz attack, the package was already reproducible.

[jcranmer]

I'll give an analogy to email and spam. A lot of effort has been spent making sure that if an email is from [e-mail], it actually came from [e-mail], giving us things like SPF, DKIM, and DMARC. And it turns out that the most eager adopters of the newest technology are... the spammers themselves! Because they don't need to lie about their email address; they can have that be completely honest, and instead resort to other tricks to mislead users as to who they are (e.g., the display name, which most email clients blindly trust and happily display).

Similarly for package managers, the biggest issues are typo-squatting or maintainer credentials compromise. And in neither case does the attacker have any incentive to take advantage of it in a way that breaks reproducibility--they can be completely honest about what they're doing. Now even if I were an attacker who had compromised a maintainer's machine, I'd still probably go for compromising the source rather than compromising the final artifact-generation process... simply because compromising the source makes the exploit live longer.

As xz shows, once you have a compromised maintainer, there's basically nothing you can do to fix it except by having someone else discover the compromise and locking out the maintainer.

[uecker]

Typo-squatting attacks are more of an issue for non-curated software collections, not so much for Debian. If you use npm or cargo or similar, then you have indeed far bigger worries. Compromising the source has the disadvantage for the attacker that it much easier to detect. Again, if you always install the newest things from an non-curated collection that may not matter much, but for something such as Debian, this increases the probability of detection a lot. One can argue that xz shows that it is possible to hide things in the source, but it also shows how much effort was needed to do this. (and the xz package was reproducible, so compromising the debian system and uploading a binary would have a high risk of detection. That this was not done can therefor not serve as evidence that binary attacks are not an issue. )

[goodpoint]

No, reproducible builds make such backdoors more difficult to sneak in together with other checks.

[eptcyka]

It makes shipping backdoors a whole lot harder, yes.

[PunchyHamster]

Unless someone spins entirely separate infrastructure dedicated just for verifying Debian packages, it doesn't.

[zaphirplane]

Hmm, it prevents Trojan binaries which is a small subset of backdoor IMHO.

Defense in depth obviously is a good thing

[aborsy]

There was perhaps no detected bug or attack. There have most likely been bugs or attacks that reproducible builds would have prevented.

[CyberDildonics]

There have most likely been bugs or attacks that reproducible builds would have prevented.

Like what exactly?

[PunchyHamster]

And you base it on what exactly ? It's "just" making sure the build process is always ordered.

If anything it will make attacker's job easier, as Ubuntu package will have same files structured exactly same way as Debian one.

[yjftsjthsd-h]

> as Ubuntu package will have same files structured exactly same way as Debian one.

As opposed to what? If Ubuntu uses the same source, of course they get the same binaries. And if Ubuntu applies patches, they'll get something different. And that's still true.

[deknos]

"mimimimi".

Those people do not care about quality in opensource at all. For longliving software this is very important.

Of course, all those javascript and kubernetes packages which are irrelevant in a few years again, might complain, but let them complain.

[ckastner]

> There was no bug or attack on Debian since 2007 that reproducible packages would prevent.

I'm reading this as a suggestion that the reproducible builds effort was an ineffective deterrent.

However, note that your observation could also be explained by the opposite: the reproducible builds effort was an effective deterrent, so nobody bothered with attempts.

> And it just ups the the contribution barrier to Debian higher

Until yesterday, the package just got flagged in the tracker, and you could either ignore it, or fix it yourself, or the kind people behind the reproducible builds effort supplied a patch themselves.

Now, you can no longer ignore it. But fixes are often trivial. Use a (stable) timestamp provided by the build, seed RNGs with some constant (instead of eg: time), etc. These are best practices anyway.

[PunchyHamster]

> However, note that your observation could also be explained by the opposite: the reproducible builds effort was an effective deterrent, so nobody bothered with attempts.

There was no attack that reproducible builds would help protect from before 2007 either.

> Until yesterday, the package just got flagged in the tracker, and you could either ignore it, or fix it yourself, or the kind people behind the reproducible builds effort supplied a patch themselves.

> Now, you can no longer ignore it. But fixes are often trivial. Use a (stable) timestamp provided by the build, seed RNGs with some constant (instead of eg: time), etc.

that's the entirety of the problem. App developers don't want to be package experts or build experts.

> These are best practices anyway.

They are not. They are best practices if you want reproducible builds. They are entirely useless waste of time if you don't care.

[yjftsjthsd-h]

> that's the entirety of the problem. App developers don't want to be package experts or build experts.

App developers and Debian package maintainers are already separate groups.

[ckastner]

> They are not. They are best practices if you want reproducible builds.

Or if you're writing a test suite, and you want failing test results to be actionable.

Or you have any other type of behavior that you'd like to reproduce somehow.

One of the first things app developers ask for in bug/issue templates are the steps to reproduce something. I wonder why you'd think that they would suddenly be opposed to the concept when thinking of a build peocess.

[PunchyHamster]

> Or if you're writing a test suite, and you want failing test results to be actionable.

The class of bugs would be extremely small as the stuff that makes build hard to reproduce are 99% of the time stuff irrelevant to runtime like some build time embedded in binary, some file metadata having different timestamp, or maybe linker putting stuff in a bit different order.

> One of the first things app developers ask for in bug/issue templates are the steps to reproduce something. I wonder why you'd think that they would suddenly be opposed to the concept when thinking of a build peocess.

I think you will find amount of people that had problems reproducing because of non-100% exact build is vanishingly small, possibly non-existent.

And that is because if you get package version and want to reproduce it, you get the package, install it and try to reproduce it. The package WILL be 100% the same as the one you got in bug report because you both downloaded the same artifact from same mirror network. You don't need reproducibility to get same binary to reproduce bug

[Atotalnoob]

That’s a big logical fallacy, I’m not sure if that’s what you want to go with