[calahad]

"Maybe there's something glaringly wrong in that particular paper, but these authors do nothing to actually demonstrate that."

He first states that the particular paper's flaw is the use of implausible statistics, then goes into more detail:

"There was also considerable silliness including claiming to have tactfully inspected the genitals of slightly fewer than 10,000 dogs whilst interrogating owners as to their sexuality (“Dog Park”),"

[slivym]

Is that silly? As an academic am I meant to be familiar with the number of dogs who frequent a particular park in a country I've never been to? As the reviewer of that paper what exactly are you suggesting, I accuse them of lying?

Academia is set up to tackle people who fabricate their results - the reputational damage would destroy most people's careers. But that mechanism is not some sort of fact-checking investigation by the peer reviewers.

Let me put the counter to you: That paper has been cited precisely 0 times according to publications website. In fact, the only references I can find to it are non-academic websites which generally trawl research for funny papers they can write jokey articles about. So what has this told us about Academia?

[foldr]

>Academia is set up to tackle people who fabricate their results - the reputational damage would destroy most people's careers. But that mechanism is not some sort of fact-checking investigation by the peer reviewers.

This is an important point, and slivym shouldn't be downvoted for making it. Many people outside academia seem to have unrealistic expectations of the peer review process. Reviewers can't, in most cases, verify experimental results. Ironically, this is especially true of the so-called "hard" sciences, where a typical experiment might take months or years of preparation and cost lots of money to carry out.

The only real protection against fabricated results, in any field, is the honesty of its practitioners. What we have here is a group of rather naive people discovering that it's easy to lie and get away with it.

[buboard]

> Many people outside academia seem to have unrealistic expectations of the peer review process

Disagree , and i think slivym has confused peer review with sloppy view, or perhaps peer review is indeed sloppy in their field. Yes, peer review is a bad system but it is not "nothing". Reviews won't redo the experiment but they may ask for a lot of work in reviews , they can be opinionated , disbelieving, and this is good, it keeps a certain baseline, and in my experience always improves the paper. You can easily tell if a manuscript that has been reviewed or first-submitted. It s not a matter of black and white: peer review sits somewhere in between but it's not black.

[foldr]

I didn't say that peer review was "nothing". I said that reviewers cannot usually verify the results of the experiments in the papers that they're reviewing.

For example, my (erstwhile) field is linguistics. Suppose that I review a paper about noun incorporation in Mohawk, and the author makes various claims about which kinds of nouns can and can't incorporate into which kinds of verbs. Not being an expert in Mohawk, I can't verify those claims. If the language in question is something less studied than Mohawk, it may be that there is no reviewer available who can verify the claims without making an impractical expenditure of time and effort.

At some point, you just have to rely on the fact that most people are not fundamentally dishonest. It's like that in any field. Reviewers work for free, and aren't going to spend months verifying the results of a complex experiment.

[buboard]

yeah i expanded on that. Re-doing the experiment is not the only way to improve the results, and it would be dangerous to claim that if you can't outright falsify a paper it should be accepted. When so much science is being published by more people than any other time, the standards should always be raised. Trust is not to be assumed imho.

[foldr]

As you, say, reviewers won't (and most likely, can't) redo the experiment. For this reason, there is no real protection in the review process against people making results up.

If you didn't trust by default, you'd never publish anything.