[clavigne]

I did peer review for a number of scientific papers that include code. Almost every time, I was the only reviewer that even look at the code.

In most cases, peer reviewers will just assume that authors claiming the "code is available" means that a) it is reproducible and b) it is actually there.

As a counter example, this recent splashy paper

https://www.nature.com/articles/s41587-021-00907-6

claims the code is available on github, but the github version ( https://github.com/jameswweis/delphi ) contains the actual model only as a Pickle file, and contains no data or featurization.

So clearly, the peer reviewers didn't look at it.

[enriquto]

that. The main task of the reviewers should be to re-run all the experiments on their own computer and check the results.

[clavigne]

re-running is definitely too much work for most scientific papers, at least in ML and computational sciences were experiments might take 1000s of core-hours or gpu-hours, but that's usually not necessary. In addition, just running the code can spot really bad problems (it doesn't work) but easily miss subtle ones (it works but only for very specific cases).

I think it's more important for reviewers to read the source, the same way one would read an experimental protocol and supplementary information, mainly checking for discrepancies between what the paper claims is happening and what is actually being done. In the above example, a reviewer reading the code would have spotted that the model isn't there at all, even though it runs fine.