[allenofthehills]

This is a common misconception, but you are actually talking about "replicability" which is "writing and then running new software based on the description of a computational model or method provided in the original publication, and obtaining results that are similar enough" [1]. Reproducibility instead refers to running the same code on the same data to get the same results [2].

[1] Rougier et al., "Sustainable computational science: the ReScience initiative" https://arxiv.org/abs/1707.04393

[2] Plesser, "Reproducibility vs. Replicability: A Brief History of a Confused Terminology" https://doi.org/10.3389/fninf.2017.00076

[jpeloquin]

It's worth noting that the two provided references disagree on their definitions of reproducibility. Plesser quotes ACM as reproducibility being tested by a "Different team, different experimental setup", consistent with use in the GP's comment, but ultimately seems to favor ignoring "replicability" and instead using "reproducibility" in combination with various adjectives. The common understanding in the physical sciences is also that merely re-running the same code is not a test of reproducibility.