[CamperBob2]

The paper should be judged on its own merits

Peer review is the process by which that judgment is made.

Criticizing peer review is a little bit like criticizing democracy. It's the worst possible way to go about doing things, except for all of the others we've tried.

[totalZero]

> Peer review is the process by which that judgment is made.

Not if you are yourself a scientist.

Peer review is a social method of establishing the validity of research, but it's not the only method. Another would be collaboration between well-credentialed investigators. Yet another is having the backing of a large-scale professional firm or institution.

Peer review is lauded by people who either (A) don't know anything about research, or (B) do research without stellar credentials or the imprimatur of a large institution. Its chief proponents are the publications and conferences themselves, whose entire business model depends upon a perception of the elevated status of peer review.

Plenty of fictitious and fraudulent claims have been validated by peer review. https://en.wikipedia.org/wiki/List_of_scholarly_publishing_s...

If you want to review the research, read it yourself and evaluate the methodology.

[sebastialonso]

It seems you have discovered the egg and the chicken problem.

What is a "well-credentialed" Investigator if it's not measured by peer review? It seems to me like an argument from authority, which is ten thousand times worse than "problematic peer review" you claim. Define well-credentialed without peer review without a cycle please.

What is "backing of a large-scale professional firm or institution"? They already exist, they are called journals, which perform, surprise, peer reviews. Other alternatives you mention are probably private organizations, you seriously believe it's less problematic than already less problematic journals? Much naivety in this argument.

Peer review has its problems, but as you've been told before, it's the best we have come up with. Your proposals are much worse than what we have.

[throwaway201103]

Peer review basically confirms that there are no obvious, glaring problems with the methodology or conclusions. It's not like the peers actually reproduce the experiments.

If you are a "brand name" researcher (i.e. already well known and respected by your peers) peer review is even less meaningful as it's pretty much a rubber stamp.

[philippoi]

My wife is a researcher and says she can often guess who the authors are when she performs reviews because it's her domain and their references and context tend to give it away. But it's still a guess because the review process is blind. I'm pretty sure it's just the journal's editorial staff that can see who's who. It's not perfect, but it's not as blatantly biased by 'brand name' stature the way you suggest.

[matham]

Isn't it expected that as scientists understand their field in greater depth, they make less mistakes and their papers would be closer to the truth and be rejected less?

It's not like many of these fields are just starting up.

[thu2111]

You'd expect that yes, but weirdly, you'd be wrong. Journal impact factor and likelihood of replication don't seem to correlate:

https://fantasticanachronism.com/2020/09/11/whats-wrong-with...

Studies that don't replicate are cited at the same rate as those that do:

https://fantasticanachronism.com/2020/09/11/whats-wrong-with...

This year I've read a lot of epidemiology papers, and sometimes their peer reviews. There's something deeply wrong with peer reviewers in this field because they often write long, detailed reviews that completely ignore glaring problems in the papers, problems that jump out to 'lay' readers on the first glance through. My guess is that there are very complicated sets of unwritten rules about the sorts of problems that are and are not legitimate to criticise in peer reviews, and problems that are a little bit too fundamental, of the form "this entire paper should be rejected out of hand", don't get given when a paper has 25 authors at prestigious universities, even if the methodology or conclusions are absurd.

[matham]

Ok, but I think your examples are specific to the social sciences where the method used is not sufficiently close to the scientific method to be reliable enough. Hence you're likely to see systemic biases in junior and senior authors alike as the field may not converge to "truth".

But in fields that the method is closer to science (e.g. physics, chemistry, neuroscience), I would expect that the overall field is converging to the truth and that senior authors will therefore be more tuned in to the best estimate of truth or how to get to it than junior authors.

[thu2111]

Yeah, but do you have a formal list of academic fields labeled as scientific by media/government but which are not actually scientific? The term "social science" doesn't cover it, as epidemiology is proving. Not many would call it a social science but the problems there are identical or frankly even worse. And what of climatology, another field where people construct complex models on relatively small datasets and can't do even small scale experiments? Is that also a social science? Clearly not.

Even in microbiology there are a huge number of papers that don't replicate.

To me it looks like the problems are general. They aren't restricted to a small set of social sciences.

[sudosysgen]

Attempting to review the research of the NAS makes it obvious that their findings are way under the standard of evidence for such a claim. They also fail to give any kind of parameters for what this kind of RF would look like, as well as fail to make specific, falsifiable, claims on what it is.

While plenty of bad claims have been validate by peer review, it doesn't mean that peer review doesn't massively reduce the number of outrageously incorrect claims.

In any case, as it stands, the NAS article wouldn't pass peer review, because it makes borderline unphysical claims without providing enough evidence, even though doing so is quite easy. As for stellar credentials or the imprimatur of a large institution, one of the authors is fairly kooky and the institute itself has a massive conflict of interest.

There's a reason even the State Department is distancing itself from this thesis.