But is it "reproducible" or reproducible? Holger still considers the debian numbers "reproducible" as we are only building things twice. To achieve proper reproducible builds the artifacts needs to be reproducible by users. User facing tools needs to be provided and I have yet to see how NetBSD provides this.
Unfortunately, nix does not produce fully reproducible builds. The build environment is portable and produced in a way that it can be repeated, but due to the limitations of the software that is being built, the builds are not binary reproducible. You can see some commentary on the nix team hoping to adopt some of the work being done by debian et al here: https://github.com/NixOS/nixpkgs/issues/9731
NixOS currently isn't there. A lot of this work is done, but much still remains. We have benefited greatly from Debian's work, though (Debian maintainers frequently come across as happy upstream participants to fix issues like this in the ecosystem, which really helps everyone!)
https://r13y.com/ tracks the progress of NixOS reproducibility; currently we're at 98.23% bit-for-bit identical for our minimal installer ISO. After that, we'll need the graphical installer, and then more of the base package set. So we've still got a ways to go.
Any program with a build system designed in such way it doesn't introduce anything beyond the source code into the binary should be.
If you build the same code on two different machines, using the same compiler, with the same options, then the generated binaries should be exactly the same.
> If you build the same code on two different machines, using the same compiler, with the same options, then the generated binaries should be exactly the same.
There is so much context that is normally embedded into a binary that this is usually not true unless explicit measures have been taken.
Two very common sources that introduce variability are time-stamps used in the build, and environment variables such as $HOME and $USER.
If you're generating or modifying source code at build time (eg. adding timestamps or build IDs) then you have violated the constraints on build reproducibility.
If you define the problem as excluding things a large percentage of real-world build systems do by default, then it's not very interesting. The interesting part of Debian's and others work here is making this work with small, unintrusive changes to such systems.
You don't even need multi-threading. In gcc we had at least one case where a key=>value data structure was keyed by memory address, causing symbols to be emitted in different order depending on ASLR, phase of the moon, or whatever.
Most compilers give no guarantees in which order they lay out the data. I love deterministic processes as much as everyone. But randomized approaches have their advantages too. And if a compiler has reasons to randomize output e.g. for speed than it’s a trade off to consider.
Why is it a bug? I write a program to download four files. I do so in parallel. Sometimes X finishes first, sometimes Y finishes first, and the files are written to disk in a different order. Why do I want to serialize this operation?
Until a build process starts naming things with timestamps, locales, etc. Just because the build is "source code only" doesn't mean it is deterministic.
That's why I wrote "it doesn't introduce anything beyond the source code into the binary". Unfortunately, I forgot to emphasize the anything.
A build process that names things with timestamps or leaks your locale into the build configuration (or doesn't pin build-time dependency versions) will make the build depend on things other than the source code (both program and build settings) you made available.
It may even be desirable for it to be non-reproductible - if, for instance, you want to use optimizations targeted to your specific system, then your build system will have to introduce the architecture information into the build process and your build will result in a unique binary that targets your own machine.
Unfortunately, if we take this definition of "anything" literally, it is impossible to build such a build system.
For example, depending on the input order, linker may produce different output. Surely you can sort the object files, but the sorted object files order is still effectively "stored" into the binary, and that's not source code.
You can only normalize such things (like in the example above, sorting), you can not eliminate them, they naturally exist.
This is actually an annoying challenge of reproducible builds. In many cases it is actually useful to have a build timestamp, git sha, or build number available for debug output from the program. I've often gone as far as embedding a sha and/or timestamp into a file on export into a tgz which allows it to be reproducible from the tarfile, although builds directly out of source control would not be.
I maintain my own build farm and tried comparing my results against the official CI server:
$ guix challenge --substitute-urls="https://ci.guix.info"
14,224 store items were analyzed:
- 4,972 (35.0%) were identical
- 265 (1.9%) differed
- 8,987 (63.2%) were inconclusive
Of the 5237 build artifacts that were available on the substitute server, only 265 (5%) differed.
All of these items can be (and have been) built entirely from source, starting with Guix' initial "binary seeds", on (probably) different hardware and kernel compared to the CI system.
I don’t think “one artifact, one vote” is a fair way to measure this.
One reason builds become irreproducible is when a build is multi-threaded, and the order in which artifacts are combined into larger ones becomes unpredictable. That problem doesn’t exist, or at least is a lot smaller, for ‘leaf’ artifacts (example: if your C compiler is single-threaded, and you run make multi-threaded, individual object files do not have the ordering problem, but libraries built from multiple object files do)
On the other hand, a single static struct with a padding “hole” that isn’t consistently written that happens to end up in lots of binaries will decrease your percentage a lot.
If you mean which distribution has 100% of its packages reproducible, probably none yet. But Arch and Debian are both making progress.