[ymse]

I think GNU Guix offers what you are after.

I maintain my own build farm and tried comparing my results against the official CI server:

  $ guix challenge --substitute-urls="https://ci.guix.info"
  14,224 store items were analyzed:
    - 4,972 (35.0%) were identical
    - 265 (1.9%) differed
    - 8,987 (63.2%) were inconclusive

Of the 5237 build artifacts that were available on the substitute server, only 265 (5%) differed.

All of these items can be (and have been) built entirely from source, starting with Guix' initial "binary seeds", on (probably) different hardware and kernel compared to the CI system.

[Someone]

I don’t think “one artifact, one vote” is a fair way to measure this.

One reason builds become irreproducible is when a build is multi-threaded, and the order in which artifacts are combined into larger ones becomes unpredictable. That problem doesn’t exist, or at least is a lot smaller, for ‘leaf’ artifacts (example: if your C compiler is single-threaded, and you run make multi-threaded, individual object files do not have the ordering problem, but libraries built from multiple object files do)

On the other hand, a single static struct with a padding “hole” that isn’t consistently written that happens to end up in lots of binaries will decrease your percentage a lot.

[ymse]

Sorry, I think my use of "artifact" here caused some confusion.

Each of these "artifacts" are actual isolated builds of complicated programs such as Chromium or GCC. The technical term is "derivation", which produce "outputs".

All of those packages can be reproduced from source now or 100 years into the future and SHOULD produce the exact same binary output. If they don't, it's a bug.

[bchallenor]

What does guix mean by "inconclusive"?

[ymse]

It means that the items could not be found on the remote server(s).