Peer review, while extremely useful in bringing out some, or many warts of a work, is not a substitute for quality control, which depends only on the authors.
> Don't expect that you, the average HN reader, knows better than reviewers with relevant backgrounds.
After what happened with the Surgisphere paper, where the first problem was the statistics (caught by post-publication comments) and only later there was the problem of fake data, the mere presence of a secret review does not mean the work is up to anything.
P.S: yes, I work in the fields of biology and pharmacology and have written and reviewed a number of papers myself.
I still think it's a fair comment that most Hacker News readers are not the audience for pre-prints. It's simply not possible for people untrained in the art of reading papers in a specific field to make reasoned judgements.
Out of curiosity, do you happen to have handy the proportion of papers submitted for publication that do not pass peer review?
I've worked in the scientific publishing field (though, admittedly, as a lay person) and remain generally unimpressed by the process. At best, I tend to see a paper having been accepted for publication as a positive signal - certainly not as a clear indicator.
It depends highly on the journal. In my field (optics) for a 'high impact' journal (which a paper claiming to show a way to curb the covid-19 pandemic would certainly qualify for) the accept rate is about 10% after multiple rounds of review/revision.
After your paper gets rejected from a top journal you apply to a mid-tier journal (usually for reasons related to impact, but this is tied to your claims that are not watertight being struck out in the review) which has about a 50% accept rate after you remove the unproven claims.
Then when you don't get accepted there you go to a 'just publish my paper' journal, which comes in 2 flavors:
The respectable move is to publish in a technical archive (nature scientific reports is a good example of this) where you only have to satisfy that the work is technically sound, so you remove all of the unsubstantiated claims (title goes from 'we cured covid' to 'statistical analysis of covid patients--results inconclusive') and publish your measurements and methods.
The less respectable way out is to publish in a journal with a fast-track review process that just accepts anything as long as you pay the fees. Usually people in the field are smart enough to ignore papers in those journals, but bean counters and media aren't.
> the accept rate is about 10% after multiple rounds of review/revision
This is part of what I'm driving at here - "accepted for publication in this journal" is not the same as "peer reviewed". Broadly speaking when someone says that a paper has been peer reviewed, they only mean that a third party has examined the paper and certified that they did not find any significant issues with the methodology. The GP didn't say "this hasn't even been accepted for publication" - they said "this hasn't even been peer reviewed yet". Those are very different things!
I would see acceptance into a top-tier journal - or even a reputable journal - as a very, very strong signal.
I'm still curious what proportion of papers meet the bar for inclusion into a journal but are filtered out by the peer review process. I suspect it's vanishingly small.
> Then when you don't get accepted there you go to a 'just publish my paper' journal [...]
Right, which is why I always take the conclusions of a paper with a grain of salt inversely proportional to the prestige of the journal in which it was published. If I'm looking at a paper in Nature, I'm as confident as is reasonable that the conclusions of the paper are correct (or at least well-founded). If I'm looking at a paper that hasn't been published in a journal of any type, then I essentially discard their conclusion and consider the data and the author's analysis (with a healthy dose of skepticism).
In this case, even as someone who is definitely not a professional in the field, I identified what I believe to be glaring methodological errors: https://news.ycombinator.com/item?id=23690529
> I would see acceptance into a top-tier journal - or even a reputable journal - as a very, very strong signal.
Not anymore, I'd say. Both The New England Journal of Medicine and Lancet were fooled by the Surgisphere scandal, and they didn't even notice the statistical errors, let alone the fabrication of the data.
Lancet took years to remove the Wakefield paper on autism and vaccines, and still has other papers with questionable statistics up.
What I consider a strong signal is good data with clear conclusions and well-outlined limitations (these are often as important as the results themselves). Peer review can help with that, but it's not a magic bullet. In that sense top tier journals can be "dangerous" because sometimes they focus on good storytelling instead of presenting the data correctly.
> Don't expect that you, the average HN reader, knows better than reviewers with relevant backgrounds.
After what happened with the Surgisphere paper, where the first problem was the statistics (caught by post-publication comments) and only later there was the problem of fake data, the mere presence of a secret review does not mean the work is up to anything.
P.S: yes, I work in the fields of biology and pharmacology and have written and reviewed a number of papers myself.