Hacker News new | ask | show | jobs
by matusp 1108 days ago
Peer review is not a certificate of truthfulness, peer review just says that a grad student reading the paper did not found anything too suspicious there and it looks like a typical paper in that field.
3 comments

Peer review, at its core, is a social consensus process. It can work, but it is structurally inclined to propagate agreement.

I wonder if it serves to reinforce this problem, as those more motivated to reinforce certain literatures or perspectives will surface those repeatedly in reviews.

There's a popular work that comes to this conclusion in its beginning chapters [0]. The basic premise is that the act of science and knowledge acquisition cyclically devolves into ideology and is then disrupted. Typically those who initially disrupt a scientific dogma are not treated well. Eventually the old guard literally dies off and the new ideas can begin to take hold.

[0] The Structure of Scientific Revolutions, T. Kuhn

> The basic premise is that the act of science and knowledge acquisition cyclically devolves into ideology and is then disrupted.

This can happen of course, but it's not really what Kuhn is talking about. Kuhnian paradigm shifts are to science what massive refactorings are to a codebase: the problem isn't that the old code was wrong (though it might also be wrong), it's that it can't be extended to meet new requirements. And while some of us are better at writing extensible code than others, no one gets it right every single time.

Not even that. I was a grad student that reviewed a paper submitted to a well-known journal from a heavyweight in my field. The paper's own data showed that the authors were reporting 95% noise. My advisor rejected the paper only to see it published a few months later (in another famous journal) after the authors removed the data that allowed us to detect the noise throughout the data.
A huge part of the problem is that the layperson is taught otherwise.
2nd grade we learned the scientific method. I don't remember peer reviewed being a thing.

Replication was critical.

Also, I've seen peer reviewed papers with atrocious stuff in them, unless I know the peer, it doesn't mean much to me. Give me 100 independent groups coming to the same conclusions. That has far more weight than a few of your friends/professors reading your paper.