[aggie]

In practice, peer review isn't even that. Most referees are not double-checking your statistical analysis. What they focus on is whether the research is interesting and has it appropriately considered relevant literature. Even that is not always done carefully.

Here is an argument that peer review is basically a failed experiment: https://experimentalhistory.substack.com/p/the-rise-and-fall...

[dadrian]

I think we're saying the same thing. You're trusting the statistical analysis done by the author. When I peer review, I check to make sure that the conclusions the paper draws are in line with what it says statistical analysis was, and I check to make sure that the analysis used is the right one, but I don't check their math. And also, I'm not a statistician, so I'm not authoritative on the full space of statistical analysis, anyway.

[nieto915]

who really has all of the hours in a day to do it too. I think it'll all come down to the journal's integrity. academia will have to learn to negotiate the premiums of their library subscriptions for better editing/curation for each journal they subscribe to. its already pay to play, might as well get your money's worth.

[kuhewa]

I'm burnt out on reviews, I did maybe ten last year but never spend less than a day on each and sometimes two. In my papers every reference fits to the best of my knowledge (knowledge informed by actually reading said cited material) and i follow best practices in statistical analyses, so I hold other papers to the same standard. Just an unsustainable standard to hold. I'd like to see journals pay statisticians to focus on methodology, I'm not sure if honourariums are a solution for peer review though. It would help me justify spending the time to do them but I reckon there would be a subset of hyperprolific reviewers half assing it more than ever to maximise the income per time spent.

[jhbadger]

And often it is impossible to replicate the results as a reviewer even without fraud being involved because at least in studies involving DNA sequences, these sequences are generally only added to Genbank (or EMBL, or whatever repository depending on country) when a paper is accepted and so reviewers don't have access to it.

[fsckboy]

In practice, peer review isn't even that. As fields of science have grown more specific, qualified reviewers are peers who are your competition, where you are already in their club or somebody who needs to be kept out. You are not going to be helped in any science that threatens their funding. There are a lot of interests vested in every instance of medical research fraud.

[ouid]

Like many human inventions, it started out good and got worse as people got better at exploiting the flaws in the institution. Those exploits eventually get bad enough that they can escalate to creating new flaws to be exploited until the whole thing is captured. Most institutions are somewhere along that path.

My personal philosophy about such things is to think of Hanlon's razor as a boundary condition. The longer an institution has been around, the more likely the incompetence is actually just well disguised malice.

[wolverine876]

It's imperfect, but what better system do we replace it with? It's easy to critique; it's hard to solve problems. Just like it's easy to find problems in any research, but hard to research and publish.

[anonylizard]

The best system we can currently observe, is in ML.

Look at stable diffusion as an example. Incredible papers, such as dreambooth, LORA, controlnet, are:

1. Published on arxiv before peer review

2. Productionised within 2 weeks of paper release (peer review not needed)

3. Community rapidly adapts tool, makes it easier to use.

4. Products built on such papers proliferate extremely rapidly within another few weeks.

In this system, peer reviews are worthless. The github code quickly demonstrates whether a technique is useful or not, and the community adoption rates replace citations as proof of a paper's power.

This is why AI art can progress at such insane rates, weeks from paper release to widespread productionisation.

Obviously, this won't work in most other domains, because there's no equivalent to mass consumer interest, open source communities, and low-cost experiments. But it does represent the ideal of an academic research paradigm.

[getoffmycase]

> products built on papers

This is the key to everything else. There is built in reproducibility and amplification of new, functional ideas in the ML community.

For the most part in life sciences, papers are published to achieve current grant aims and write future grants that will be funded. You can be an academic and love your research area and be ultra-passionate about it, but at the end of the day, grants are the end product that you are working for.

Your science does not have to work or be replicated, all you need to do is publish papers that make grant reviewers think you are reliable enough to not waste federal grant money. Nobody on the grant review board has time to look carefully to see if you papers are not fraudulent.

Let’s look at physics on the early 20th century, which had progress even faster than today’s machine learning research. Massive upheavals and rapid progress in understanding our world, including 4 different models of the atom (including the most correct one we still use today) and general relativity. What’s the difference to today’s life sciences? At the important epicenters of the day, working in the field was 1) contributing new observations, 2) directly testing somebody else’s theories with an experiment.

In today’s world, very rarely will somebody contribute new observations without an underlying motivation (get new grant money, advance current grant claims). And nobody has the time or resources to test other people’s ideas with new experiments. Why? Cause research is expensive and you would need a grant to fund a replication. And no government body funds those grants.

Disclaimer: there’s people in life sciences in some fields doing good work.

[jostmey]

The peer-review system is antiquated, developed before the internet and powerpoint presentations. Anything that facilitates interaction between authors and other researchers will be a vast improvement.

Peer-review doesn't catch fraud and is sometimes a political process. I've found the best corrections I've gotten is after posting pre-prints to online forums. I suggest that commenting on pre-prints is better than peer-review pre-publication. I imagine a ranking system could highlight comments from trusted reviewers. Studies that no one wants to review were probably never going to be read anyway, and so there was never any reason to review these studies anyway.

[a_bonobo]

Yeah that's post-publication peer review, which I tend to gravitate towards myself; un-peer-reviewed papers in my field, posted on biorxiv, are generally of pretty high quality. (this might change as preprinting becomes a fully established route in biology)

Any peer review added to that might improve some things, but is it worth it to add ~6 months to the publication timeline for a marginally improved manuscript? I'd say no.

[BurningFrog]

Widespread peer review is only ~50 years old.

Science did just fine before it, as it will after it's phased out.

[jhbadger]

Well, it was around but inconsistently applied. Einstein famously got angry when his work on gravity waves in 1935 was sent out to review ("I sent my paper to be published, not to be reviewed!"), and Watson & Crick's 1953 paper was going to be sent out to review but their boss, Nobel Laureate Laurence Bragg, phoned up the editor of Nature and said they needed to get it published ASAP.

[a9h74j]

Forty years ago there was already this sentiment, as best as I can recall the quote: "It is well known that as the scale of the research grows, peer review becomes less effective."

[naasking]

> It's imperfect, but what better system do we replace it with?

Pre-registered trials and/or arxiv + open science.

[TheSpiceIsLife]

Science theatre