[hijodelsol]

I came here to say something similar. As someone who works in a field that applies machine learning but is not purely focused on it, I interact with people who think that arXiv is the only relevant platform and that they don't need to submit their work to any journal, as well as people who still think that preprints don't count at all and that data isn't published until it's printed in an academic journal. It can feel like a clash of worlds.

I think both sides could learn from the other. In the case of ML, I understand the desire to move fast and that average time to publication of 250-300 days in some of the top-tier journals can feel like an unnecessary burden. But having been on both sides of peer review, there is value to the system and it has made for better work.

Not doing any of it follows the same spirit as not benchmarking your approach against more than maybe one alternative and that already as an after-thought. Or benchmaxxing but not exploring the actual real-world consequences, time and cost trade offs, etc.

Now, is academic publishing perfect? Of course not, very very far from it. It desperately needs to be reformed to keep it economically accessible, time efficient for both authors, editors and peer reviewers and to prevent the "hot topic of the day" from dominating journals and making sure that peer review aligns with the needs of the community and actually improves the quality of the work, rather than having "malicious peer review" to get some citations or pet peeves in.

Given the power that the ML field holds and the interesting experiments with open review, I would wish for the field to engage more with the scientific system at large and perhaps try to drive reforms and improve it, rather than completely abandoning it and treating a PDF hosting service as a journal (ofc, preprints would still be desirable and are important, but they can not carry the entire field alone).

[bonoboTP]

Simply anticipating basic push backs from reviewers makes sure that you do a somewhat thorough job. Not 100% thorough and the reviews are sometimes frivolous and lazy and stupid. But just knowing that what you put out there has to pass the admittedly noisily gatekept gate of peer review overall improves papers in my estimation. There is also a negative side because people try to hide limitations and honest assessments and cherry pick and curate their tables more in anticipation of knee jerk reviewers but overall I think without any peer review, author culture would become much more lax and bombastic and generally trend toward engagement bait and social media attention optimized stuff.

The current balance where people wrote a paper with reviers in mind, upload it to Arxiv before the review concludes and keep it on Arxiv even if rejected is a nice balance. People get to form their own opinion on it but there is also enough self-imposed quality control on it just due to wanting it to pass peer review, that even if it doesn't pass peer review, it is still better than if people write it in a way that doesn't care or anticipate peer review. And this works because people are somewhat incentivized to get peer reviewed official publications too. But being rejected is not the end of the world either because people can already read it and build on it based on Arxiv.

[bjourne]

I really am not sure about that: https://biologue.plos.org/wp-content/uploads/sites/7/2020/05...

The problem is that "optimizing for peer-review" is not the same thing as optimizing for quality. E.g., I like to add a few tongue-in-cheeks to entertain the reader. But then I have to worry endlessly about anal-retentive reviewers who refuse to see the big picture.

[bonoboTP]

Currently a kind of rule of thumb is that a PhD student can graduate after approximately 3 papers published in a good peer reviewed venue.

If peer review were to go away, this whole academic system would get into a crisis. It's dysfunctional and has many problems but it's kinda load bearing for the system to chug along.

[mitthrowaway2]

Maybe their institution should evaluate whether their papers pass muster? It's the one conferring the degree.

[DANmode]

No hard rule, no crisis.

Maybe we can go back to very opinionated “true” academia,

where there are institutional gatekeepers,

but they mostly get it right on who to award (and not),

vs the current game of

“whoever plays ball with funding sources the best = the best academic”,

which is obviously bullshit.

[vkou]

You'll still need to convince the purseholders to pay you, and they'll want some objective metric to measure your output, and whatever metric they pick will be gamed.

[DANmode]

The point of my comment was,

in much earlier institutions of knowledge and excellence,

the only transparent metric was whether or not they approved you.

[pie_flavor]

You may have delivered value in peer review, but on the whole, peer review delivers negative value. https://www.experimental-history.com/p/the-rise-and-fall-of-...

The arXiv vs journal debate seems a lot like 'should the work get done, or should the work get certified' that you see all over 'institutions', and if the certification does not actually catch frauds or errors, it's not making the foundations stronger, which is usually the only justification for the latter side.

[fc417fc802]

Can't say I agree with that position.

Responding largely to the linked article, you can't just ignore the massive increase in funding and associated output that occurred. Scaling almost any system up will be expected to result in creative new failure modes. It's easy to observe that a system isn't great and suppose that removing it would improve things but this very often isn't the case. Democracy is one such example.

There's also the publishing ecosystem that developed around the increased funding. It isn't clear to me why any blame (if it's even valid, see preceding paragraph) should be laid at the feet of the practice of peer reviewing publications rather than such an obviously dysfunctional institution.

Even if we accept the way in which publications have been undergoing peer review to somehow be the root of all evil (as opposed to the for profit publication of taxpayer funded work) - there's more than one way to go about it! A glaringly obvious problem, mentioned in the linked article yet not meaningfully addressed that I saw, is that peer reviewers aren't paid. If this was a compensated task presumably it would be performed much more rigorously. Building inspectors aren't volunteers and they seem to do a good enough job.

[StableAlkyne]

I've noticed it's field dependent. Some fields don't really feel much need to publish in a real journal.

Others (at least in chemistry) will accept it, but it raises concern if a paper is only available as a preprint.

[observationist]

What's the value of academic publishing over the arxiv model of freely publishing, free access, and a global, vigorous discussion across a wide range of platforms, with experts, researchers, amateurs, institutions, and the peanut gallery all having the opportunity to participate?

What possible value does a journal like Nature, for example, bring to the table by claiming a paper for themselves and charging people for it, given the alternative?

I don't see any value there. Maintaining an exclusive clique by using artificial scarcity while coasting on the dregs of reputation remaining to a once prestigious institution is what a lot of these journals are doing.

The world has changed. There's no need for that sort of pay to play gatekeeping, and in fact, the model does tremendous damage to academic and intellectual integrity. It allows people to get away with fraud and it makes the institutions motivated to hide and cover it up so as to not damage their own reputations by admitting anything slipped by them.

If you contrast the damage done by journals, with regards to suppressed research, gatekept access, money taken from researchers and readers alike, against the value they might plausibly provide, the answer is clear.

They're not needed anymore. The AI era, since 2017, has thoroughly demonstrated that journals are materially incapable of keeping up, that they're unable to meaningfully contribute to the field, and that their curation or other involvement has no effective practical value. The same is true for other fields, but everyone involved wants to keep their piece of the grift going as long as possible.

We don't need them, anymore. I suspect we never did.

[jltsiren]

The value is the ability to do science as a career without being independently wealthy.

Politicians, administrators, donors, and taxpayers don't want scientists deciding on their own how to spend the money. They want control over what gets funded. They want funding decisions with justifications they can understand. But they don't understand the science itself, so they need "objective" metrics to support the decisions. And because those metrics matter, people will inevitably game them.