[cherryteastain]

As a (former) reviewer at 5 journals, I disagree first and foremost with the notion that

> We need some process to gate-keep.

Journals, when print was the medium through which academic research was disseminated, had to gatekeep because there were practical considerations regarding how many articles they could put in each issue. With online repositories like arxiv, this is hardly a concern anymore.

Someone putting a crap article on arxiv does not hurt anyone else, and I'm saying this as a person who recommended tons of articles to be rejected because they had atrocious grammar/spelling issues. Worst case, it gets 0 attention and is ignored by the research community.

Something not being published in a journal/conference proceedings clearly does not prevent it from drawing tons of research attention, as we saw in numerous cases like the Adam optimizer [1].

Which brings us to the second point: what even is the purpose of a journal now? The answer is that the sole function of a journal now is gatekeeping, with the presupposition that, as you observed

> The venue of publication is a good signal, whether the time to read a paper is well-spent

Except, well, top journals have tons of articles that get 0 citations too. Clearly the filter fails at this purpose as well. So, why gatekeep at all then? Because if we did not have some exclusive prestigious journal, the plebs would not be separated from the esteemed titans of academia with the biggest grants, most prestigious scholarships and diplomas from the most famous universities.

The only reason we need to gatekeep today is to feed the academic prestige and politics machine. If you care about the science, upload the goddamn PDF to arxiv , tell your colleagues about your research at a conference and let the scientific community decide on whether your idea is interesting.

[1] https://arxiv.org/abs/1412.6980

[Al-Khwarizmi]

The Adam optimizer was published at ICLR, a top conference in machine learning. So, basically, analogous to a peer-reviewed journal for the purposes of this discussion. ML (and some other subfields of computer science) have the particularity that the really comptetitive gatekeeping happens even more in conferences than in journals.

That said, there definitely are very relevant papers that are not published in any peer-reviewed venue. A good example is "Language Models are Unsupervised Multitask Learners" (the GPT-2 paper, which I would argue started the whole generative LLM revolution). But I think if you look for this kind of papers, you will find something in common to all of them: they are by very well known researchers, elite institutions or influential companies. That's why people went out of their way to read them even if they were posted somewhere without peer review.

If you removed peer review and just relied on posting to arXiv or similar, new researchers, or researchers from less known institutions, would have no chance at all to make an impact. It's peer review that allows them to be able to submit to a top journal, where the editor and reviewers will read their paper, and they can get a somewhat fair chance.

PS: I don't really like the peer review system that much either. It's just that every alternative that I have seen proposed so far is worse.

[cherryteastain]

> The Adam optimizer was published at ICLR, a top conference in machine learning.

Fair, must have misremembered that one.

> If you removed peer review and just relied on posting to arXiv or similar, new researchers, or researchers from less known institutions, would have no chance at all to make an impact.

I disagree on this one. I did my PhD at an institution that ranks in the top 10 in the most well known university rankings, and I distinctly remember that one time when I was submitting a manuscript to a prominent journal in my field, got some reviews back which weren't positive yet were quite valid criticisms, and my professor told me not to worry because the editor is his buddy and my manuscript will get published for sure.

When that sort of "scratch my back and I'll scratch yours" culture exists in journals I don't see how peer review can be an equalizer. It just means everyone who publishes at a journal, including the less well esteemed ones, can claim they went through the rigor of peer review. Of course, we all know peer review is just a vibe check and is actually not that rigorous at all, and besides no one cares unless you published in a prestigious journal anyway. The less revered journals exist to collect $5k in open access fees for the publisher in return for hosting a pdf at the marginal cost of maybe a cent a year.

[dopu]

It seems to me that something like eLife's model is the best solution to this [0]: you still have a minimal amount of curation, but generally if a paper is written well enough and within the field it won't be desk rejected. Then, it gets published on the site and sent off for peer review. Peer reviewers assess how sound the paper is and pass a judgement which readers can view, as well as provide some recommendations to the authors make it stronger. The authors can then either revise the paper, or do nothing at all. In either case, papers don't languish in reviewer hell and the larger scientific community gets to see it.

[0] https://elifesciences.org/inside-elife/66d43597/elife-s-new-...

[jpfr]

Exactly, everybody can publish on arXiv. And there are enough semi-predatory journals/conferences which basically accept everybody. Especially since the LLM can rewrite any paragraph nicely.

So the role of journals and conferences is not to prevent "the word getting out". It is to provide value by a curated list of on-topic and high-quality publications.

So no need to wade through tons of crap. Especially for PhD students who might take more time to detect crap as such.

In my experience, the publications at the good venues get a lot more eye-balls and by consequence citations. So there seem to be a lot of people who like this role as a "filter" for what to focus on.

[xipho]

> Someone putting a crap article on arxiv does not hurt anyone else

> ... only reason we need to gatekeep today is to feed the academic prestige and politics machine

This to me says you have may not have experienced some parts of the (long-term) research process. It suggests that you have infinite resources to filter out noise, which is probably not the case. It suggests you're willing to spend a lot more time figuring out why something doesn't quite make sense, rather than get to the heart of the problem, while this is fine in many cases it sucks when you're hot on the trail of something interesting, and you're slammed by a million twisty paths full half-baked hot-takes.

We need to filter ("gate-keep" is pretty inflammatory term) information and processes so that we don't have 12 different screw types with 12 different electric screwdrivers, instead of "just" 6 (sigh). We need to come to consensus and that means some things go in and something are left out. We need many mechanisms to filter.

> tell your colleagues about your research at a conference and let the scientific community decide on whether your idea is interesting.

All of these things feel like filters, when does a filter become a gate: colleagues - i.e. not everybody, but some selected few, who and how?; conference - filter (well, gate!); scientific community - != your baker's community; decisions directed by you, not on my own (i.e. a pointer to my paper) - filter.

[Edit for formatting, sort of.]

[cherryteastain]

Conference does not imply gatekeeping. There are many conferences out there which accept almost all submissions.

As for the "scientific community" being a filter, there is a difference between "elite" researchers being the ultimate arbiters of scientific truth via their positions in the editorial teams of journals versus everyone being allowed to publish on open platforms like arxiv and bad ideas/quackery being filtered out naturally.

Because the former is what makes or breaks a scientist's career, grad students and postdocs hyper optimize to publish at prestigious venues, as opposed to optimizing for doing science. These two are aligned only sometimes.

Per Goodhart's law: "When a measure becomes a target, it ceases to be a good measure." [1]

[1] https://archive.org/details/ImprovingRatingsAuditInTheBritis...

[xipho]

> Conference does not imply gatekeeping.

Nor do journals.

I think it's straightforward to make an argument that many of today's conferences are as bad as journals, accepting submissions is only one way conferences filter. IRL they are prohibitively expensive to enter, let alone attend (but again, see "Zoom"), and therefor eliminate all but the elite, they are run by commercial entities in all cases with more than ~200 people, they are more or less required venues for networking and therefor selling yourself for a tiny chance at academic permanence, they give plenaries to elites (filtering to one voice), they have special symposia by invite only, with other submissions dumped to inaccessible parallel sessions (which one will you choose to see?), the submissions you make are published in much more ephemeral ways, and tend to be more difficult to discover in the long term, making the event important but the research not as much (at least in my experience), etc.

[tensor]

Yes, there are many scam conferences that don't do any peer review. They are a huge problem. They waste researchers time and money and exist only to extract dollars from people, not to advance science.

[thfuran]

>Someone putting a crap article on arxiv does not hurt anyone else,

But having a browsable collection of the verified non-crap articles on any given topic helps most everyone working in that area.