[janosdebugs]

No, the question is: how did peer review not catch it? I have the impression that reviewers don't have the time or incentive to give papers more that a cursory review. Independent of this case, a great many papers are published where the only "proof" is a user study or survey with an extremely low number of participants, but it still gets published. Many papers don't publish their datasets and don't contain enough detail to try and replicate their results.

There should be a real incentive/compensation for reviewing properly and real consequences if a paper gets retracted for reasons that should have been caught in review.

In this case it's fortunate that it did get found out in the end.

[phdelightful]

Your impression is correct. Peer review would never catch this. Peer review basically assumes the counter party is operating in good faith, and as a result a thorough peer review basically is the following:

* is the treatment of existing work semi-thorough (even experts don’t know everything) and fair?

* are the claims novel w.r.t the existing work? If not, provide a reference to someone who has already done it.

* can you understand the experiments?

* do the experiments and their results lead to the conclusions claimed as novel?

* does the writing inhibit understanding of the technical content?

No peer review I have ever seen or done would catch anything but the most egregious bug of this nature.

[BeetleB]

Can you describe how you would have expected peer reviewers to catch this?

[janosdebugs]

I am not an expert in statistics, but i have read quite a few papers in my area (IT/Kubernetes/etc) that had an obviously faulty methodology if you have experience at all. Reading the other comments, this software should have never been used in this manner, which it seems to me someone who is hopefully well-versed in this area should have caught. Then again, the reviewers may have had very little experience in this area. (This happened to me, when the review came back the reviewers admitted themselves in the form that this was the case.)