[Barrera]

I was expecting the author to define "real peer review," but didn't see that. The best approximation is probably gleaned from the conclusion:

- integration of preprint servers and alt metrics

- tweaking incentives to review

- making comments on papers public

- use of software to detect fraud

- directing resources specifically to improving peer review

The bigger problem is that the author doesn't seem to actually zero in on the problem peer review is supposed to solve today. The author notes that peer review really got going in the 1970s as a way to filter content flowing to overwhelmed editors. But the emergence of the internet largely nullifies that problem. Wide distribution of scientific information no longer requires scientific publishers.

The real problem is the ways in which science funding, journals, and peer review have become intertwined, with publishers playing the role of bankers in this economy. This problem is cultural, not technical. It's a historical relic and it increasingly does not serve science well.

So, what is the actual problem that journal-supervised peer review is supposed to solve in the age of the internet?

[lacker]

Nowadays, journals and peer review solve the problem of, people need to make hiring decisions and funding decisions. But these decisionmakers don't have enough technical expertise or time to evaluate all the papers from all the applicants.

The decisionmakers do have enough time to learn which are the most prestigious journals in the field. So, they can pick the people with the most papers submitted to prestigious journals, or at least use that to filter applicants down to a short list for closer examination.

[cryptonector]

It sounds funny, but it's true.

[pessimizer]

> It's a historical relic and it increasingly does not serve science well.

It's also pretty devastating evidence that the world is not going to improve in any sort of an organized way if the experts that we would expect to lead the effort for a rational world can't clean their own house. It's hard to trust academic systems to design ways to improve society when the academic system is built around an irrational base in journals.

An academic system that exhibits the same shitty array of characteristics as every other corrupt status quo institution doesn't give me a lot of hope for everything else.

[moffkalast]

I don't think anyone expects academics to fix anything, that's what we have politicians for, hah!

[fragmede]

Politicians get voted in, so they answer to the newspapers but in case that’s not funny enough, we tried just having an all powerful person in charge of things in the past but it turns out that works even even worse.

[watwut]

I dont know why would one expected academics and scientists to "lead the effort for a rational world" or be especially good at organizing or leading. It makes sense to expect them to be good at math, physics, sociology and what not.

[bryanrasmussen]

>Wide distribution of scientific information no longer requires scientific publishers.

Wide and voluminous distribution of bad information requires filters to extract the good, peer review has some form of filtering functionality, although I wouldn't say it is great I think it would probably be better than the filter that a Facebook or Twitter of Science would provide (or just Facebook or Twitter if you don't like the 'of science' locution)

[thayne]

I think having a public forum for publishing and reviewing is valuable, and then having journals or something similar, that produces curated selection of published science based on that public review.

[Fomite]

"But the emergence of the internet largely nullifies that problem. Wide distribution of scientific information no longer requires scientific publishers."

I'm not at all convinced that this is true - people have been saying it for a long time, and it's not manifested itself in a particularly compelling fashion yet.

[impendia]

I'm in mathematics, and this is very much true in my field. Papers can be, and nearly always are, widely distributed on the Internet (arXiv) before formal publishing. They are widely read, cited, and built upon.

The publishing industry survives because researchers need to put "Published in Journal X" on their CVs. The peer review process also can lead to incremental improvements, and will occasionally catch major errors, but at this point everything other than the stamp of approval is secondary.

[atwood22]

Arxiv?

[Fomite]

Arxiv, and it's various follow-ons - I think I was one of the first submissions to medRxiv, are excellent as far as they go, but if we take the COVID-19 pandemic as a stress test, they still have a long way to go.

Beyond that, their curation is...lackluster at best, even if we're not talking about gatekeeping, filtering, etc. but just "What if I want to do something other than key word search?"

I find unexpectedly interesting articles all the time reading journals. I have yet to do so for a preprint server unless someone sends it my way, or it ends up on my Twitter feed.

[impendia]

> I find unexpectedly interesting articles all the time reading journals.

Interesting. In math, I don't think anyone "reads journals", but lots of people check the arXiv frequently to see what has been posted.

[Fomite]

Math is probably the furthest along this particular pathway - which is both a good sign for this being able to work, and also not generalizable.

[theaeolist]

Journals can still do curation of the stuff that's on preprint archives. Some already do. It's a lot less friction than the current model.

[civilized]

> So, what is the actual problem that journal-supervised peer review is supposed to solve in the age of the internet?

Instead of deciding what gets published, peer review should help decide what gets noticed.

[badpun]

Journals still play the role of quality filters/QA, which is very important. While your typical Internet-only journal is going to be like Alibaba, a top journal like Nature is more like Harrods - you're not going to waste money by browsing through trash there.

[Pulcinella]

I would not even say those things would count as “real peer review.” Peer review is supposed to involve replication. Unfortunately that almost never happens these days.

[robotresearcher]

I've been a working scientist for three decades and have never heard that position before.

Are you going to delay publication of results from the LHC or JWST until someone has built another one?

The traditional unit of replication is a published paper, not a submitted paper.

A peer review takes positions on these things:

- does the paper correctly describe the state of the art, with appropriate references?

- does the paper claim and explain a novel contribution?

- is the methodology sound?

- do the claimed results justify the claimed contribution?

- does the paper clearly communicate to its audience?

That's it.

Replication will be attempted later if the audience cares about the results.

[stonogo]

What gives you that impression? Replication is a too-often neglected aspect of the scientific process, but I've never heard anyone ascribe it to the peer review aspect.