Hacker News new | ask | show | jobs
by epistasis 104 days ago
For non-specialists, I think the most important view on papers is to not view them as nuggets of truth, but communications of a group of people who are trying to establish truth. No single paper is definitive!

Peer review is an important part of scientific publication, but it's also important for the general public to not view peer review as a full vetting. Peer reviewers look for things like reproducibility of the analysis, suitability of the conclusions given the methods, discussions of the limitations of the data and methods, appropriate statistical tests, correct approval from IRBs if there are humans or animals involved, and things like that. For many journals, the editors are also asking if the results are interesting and significant enough to meet the prestige of the journal.

Peer review misses things like intentional fraud, mistakes in computations, and of course any blind spots that the field has not yet acknowledged (for example, nearly every scientific specialty had to rediscover the important of splitting training and testing datasets for machine learning methods somewhat on their own, as new practitioners adopted new methods quickly and then some papers would slip through at the beginning when reviewers were not yet aware of the necessity of this split...)

Any single paper is not revealed truth, it's a step towards establishing truth, maybe. Science is supposed to be self-correcting, which also necessitates the mistakes that need correction. Climate science is one of the fields that gets the most attention and scrutiny, so a series of papers in that field goes a long ways towards establishing truth, much more so than, say, new MRI technology in psychology.

1 comments

Sometimes reviewers also look for whether the paper cites enough of their own papers, who is publishing it (regardless of whether the review is supposed to be anonymous or not), whether it clashes with a paper they're about to publish... science is just as full of politics and corruption (if not more) as any other field.
I almost added "place the research into the context of other relevant research" as another way of saying "cite enough of the peer reviewer's papers" but fair enough.

I'm not sure if science has as much corruption as other fields, but it definitely has politics. PIs get to their position without the typical selection process for leadership that happens in most larger orgs, so there's more fragile and explosive personalities than I find in other management/leadership positions.