Bizarre. How do people make such big, splashy findings that can mess with people’s sense of optimism about science and innovation, without doing the simplest types of checks on their data and methodology.
No, the question is: how did peer review not catch it? I have the impression that reviewers don't have the time or incentive to give papers more that a cursory review. Independent of this case, a great many papers are published where the only "proof" is a user study or survey with an extremely low number of participants, but it still gets published. Many papers don't publish their datasets and don't contain enough detail to try and replicate their results.
There should be a real incentive/compensation for reviewing properly and real consequences if a paper gets retracted for reasons that should have been caught in review.
In this case it's fortunate that it did get found out in the end.
Your impression is correct. Peer review would never catch this. Peer review basically assumes the counter party is operating in good faith, and as a result a thorough peer review basically is the following:
* is the treatment of existing work semi-thorough (even experts don’t know everything) and fair?
* are the claims novel w.r.t the existing work? If not, provide a reference to someone who has already done it.
* can you understand the experiments?
* do the experiments and their results lead to the conclusions claimed as novel?
* does the writing inhibit understanding of the technical content?
No peer review I have ever seen or done would catch anything but the most egregious bug of this nature.
I am not an expert in statistics, but i have read quite a few papers in my area (IT/Kubernetes/etc) that had an obviously faulty methodology if you have experience at all. Reading the other comments, this software should have never been used in this manner, which it seems to me someone who is hopefully well-versed in this area should have caught. Then again, the reviewers may have had very little experience in this area. (This happened to me, when the review came back the reviewers admitted themselves in the form that this was the case.)
Will they own up to it and retract their broken paper that's eroding people's confidence in funding science at the highest levels? This has been an incredibly widely read and influential paper already.
I have my doubts that the authors will accept that their paper is bogus.
Particularly because the lead author landed a faculty position at a good institution based exclusively on this junk paper.
Trash comment.
1st. Splashy often comes from the media, not the scientists
2nd. One of the ways we discover problems with data is by plotting. When the plot library has a bug that hides a problem, well shit.
3rd. They did check their own findings multiple ways. Mistakes happen. The biggest critics of scientific mistakes are often those that have never done science themselves. Its easy, and its a cheap play.
Genuine question: you don’t think that outlier analysis (with moderate diligence) would have picked up the error in the original study, despite the plotting library issue? Just look at this link - this wasn’t detectable with any type of cross-validation? https://arxiv.org/html/2402.14583v1/x1.png
There should be a real incentive/compensation for reviewing properly and real consequences if a paper gets retracted for reasons that should have been caught in review.
In this case it's fortunate that it did get found out in the end.