|
|
|
|
|
|
by foldr
2818 days ago
|
|
|
>You're welcome to suggest a better example ;) You mean, I'm welcome to find supporting evidence for your argument? Shouldn't that be your responsibility? >I feel that publishing fake results should be as hard as possible. Making it "as hard as possible" would mean using almost all of the world's resources to try to stop fake results being published. If you really want to make the review process more rigorous, you need to present a concrete plan specifying (a) who's going to do the work and (b) who's going to pay for it. If you can't do that, then what makes you so sure that the review process isn't already as rigorous as it can reasonably be, given the reality of human frailty and limited resources? |
|
|
Fair enough.
I had some trouble finding the original better example I had mentioned earlier ("I recall a study looking at such impossible aggregate statistics leading to several retractions of articles whose data had been made up outright."), but while trying to find it, I stumbled on another.
Example 1: https://andrewgelman.com/2018/08/21/scandal-isnt-whats-retra...
A study gets published finding a large effect, large enough to cause a replication attempt: "The effort to replicate the original study was successful in everything except the creation of the PSU-level structural stigma variable."
The suspected reason for this replication failure (imputation of missing data) turns out to be wrong when the original authors have someone check their code for data analysis, which is found to contain an error.
If that code had been checked during peer review (or at least afterwards, by including it with the publication), the effort would have been less than a full-blown replication attempt.
Example 2: https://medium.com/@jamesheathers/the-grim-test-a-method-for...
(Which is what I was referring to earlier.)
Simply checking reported means and sample sizes for consistency revealed mathematically impossible results in 50% of tested papers.
He goes to great lengths to stress that such inconsistencies don't necessarily imply fraud (some are honest mistakes), but the behavior of some of the contacted authors when asked for their data appears very sketchy.
Again, if there were a culture of looking at raw data and inspecting analysis code during peer review, those studies reporting obviously incorrect results would not have been published, saving everyone who relies on such studies a lot of trouble.
So, who's going to do that work and who pays for it? I'd be surprised if I managed a working plan on the first attempt, but here's my proposal:
- Authors prepare their data and code together with instructions in such a way that an expert in their field can work with them without having to ask the authors for additional information. It should be attached to the paper as supplementary material. If the data is privacy-sensitive, it should at least be made available to reviewers to check that the results follow from the data. Who pays for it: whoever pays the authors to be writing papers in the first place.
- Reviewers do that sanity-check of running the code on the data to verify that the instructions are complete and the results match what is reported in the paper. They scrutinize the code to the level they'd apply to a methodology section. Who pays for it: the readers of the published paper, since they benefit from not having to do the peer review themselves when they just want to use the results.
Maybe that's unrealistic and the review process is
> already as rigorous as it can reasonably be, given the reality of human frailty and limited resources
but that would be sad.