[foldr]

>Academia is set up to tackle people who fabricate their results - the reputational damage would destroy most people's careers. But that mechanism is not some sort of fact-checking investigation by the peer reviewers.

This is an important point, and slivym shouldn't be downvoted for making it. Many people outside academia seem to have unrealistic expectations of the peer review process. Reviewers can't, in most cases, verify experimental results. Ironically, this is especially true of the so-called "hard" sciences, where a typical experiment might take months or years of preparation and cost lots of money to carry out.

The only real protection against fabricated results, in any field, is the honesty of its practitioners. What we have here is a group of rather naive people discovering that it's easy to lie and get away with it.

[buboard]

> Many people outside academia seem to have unrealistic expectations of the peer review process

Disagree , and i think slivym has confused peer review with sloppy view, or perhaps peer review is indeed sloppy in their field. Yes, peer review is a bad system but it is not "nothing". Reviews won't redo the experiment but they may ask for a lot of work in reviews , they can be opinionated , disbelieving, and this is good, it keeps a certain baseline, and in my experience always improves the paper. You can easily tell if a manuscript that has been reviewed or first-submitted. It s not a matter of black and white: peer review sits somewhere in between but it's not black.

[foldr]

I didn't say that peer review was "nothing". I said that reviewers cannot usually verify the results of the experiments in the papers that they're reviewing.

For example, my (erstwhile) field is linguistics. Suppose that I review a paper about noun incorporation in Mohawk, and the author makes various claims about which kinds of nouns can and can't incorporate into which kinds of verbs. Not being an expert in Mohawk, I can't verify those claims. If the language in question is something less studied than Mohawk, it may be that there is no reviewer available who can verify the claims without making an impractical expenditure of time and effort.

At some point, you just have to rely on the fact that most people are not fundamentally dishonest. It's like that in any field. Reviewers work for free, and aren't going to spend months verifying the results of a complex experiment.

[buboard]

yeah i expanded on that. Re-doing the experiment is not the only way to improve the results, and it would be dangerous to claim that if you can't outright falsify a paper it should be accepted. When so much science is being published by more people than any other time, the standards should always be raised. Trust is not to be assumed imho.

[foldr]

As you, say, reviewers won't (and most likely, can't) redo the experiment. For this reason, there is no real protection in the review process against people making results up.

If you didn't trust by default, you'd never publish anything.

[yorwba]

> For this reason, there is no real protection in the review process against people making results up.

You can still sanity-check the results, even without redoing the experiment. For example if the average agreement to a certain question on a 1-5 scale among a cohort of 10 people is reported to be 3.26, you might want to ask for the raw data, because that average is only possible with fractional answers.

I recall a study looking at such impossible aggregate statistics leading to several retractions of articles whose data had been made up outright.

Similarly, when someone claims that "four men watched 2,328 hours of hardcore pornography over the course of a year and took the same number of Implicit Association Tests", you might realize that 2328 hours/(4*365) > 1 hour 36 minutes per day; and ask for the titles and duration of the porn allegedly watched, just to make sure that this extremely onerous experiment has actually been performed.

Note that the paper about that "experiment" was not accepted , but at least one reviewer actually recommended less data ("My first piece of feedback on how to make this hybrid article work is that they should remove the quantitative data."), perhaps due to a misunderstanding of sample sizes ("It makes no sense to undertake quantitative analysis for four people – when you flatten the detail out of a sample of four you’re not left with anything interesting.") — the real sample size is at least 2328.

I realize that peer review mostly doesn't operate at that level of scrutiny, but maybe it should. Checking the raw data requires slightly more work of both reviewers and honest authors, but increases the workload of dishonest authors from "make up a few numbers" to "make up as many numbers as if'd actually done the work and don't introduce statistical anomalies", shrinking the gap to "actually do the work".

So even though you need to trust authors a little, it's certainly possible to trust less. There is no perfect protection against academic dishonesty, but there could be better protections.

[foldr]

> Similarly, when someone claims that "four men watched 2,328 hours of hardcore pornography over the course of a year and took the same number of Implicit Association Tests", you might realize that 2328 hours/(4*365) > 1 hour 36 minutes per day; and ask for the titles and duration of the porn allegedly watched, just to make sure that this extremely onerous experiment has actually been performed.

I don’t see the point. The authors could easily respond with a long list of porn titles. (And as an unpaid reviewer with lots of real work to do, are you going to bother verifying every title in the long list?)

>Note that the paper about that "experiment" was not accepted

Then it's not a very good example to base your argument on.

>the real sample size is at least 2328

You’re both wrong. You can’t treat 2328 observations from 4 subjects the same way as 2328 observations from 2328 subjects (see e.g. https://en.wikiversity.org/wiki/Advanced_ANOVA/Repeated_meas...)

More generally, virtually no-one understands statistics. Every field where statistical analysis is used routinely publishes papers that use bad statistical methods.

>I realize that peer review mostly doesn't operate at that level of scrutiny, but maybe it should.

What does the “should” even mean here? Do you think that reviewers who work for free “should” do even more work than they do already? Or that journals “should” force reviewers to do this (even though they have no mechanism for doing so)? There are practical limits to the amount of scrutiny any given paper can be subject to. It would suck if we needed to spend more time reviewing papers just because a bunch of assholes keep trying to get fake papers published.

>it's certainly possible to trust less.

Not really. You don't seem to realize that more scrutiny during the review process would require real people to give up more of their real time for free. You can't just snap your fingers and make that happen.