[Vinnl]

I volunteer for a project [1] where the idea of "cutting out the middle man" is taken even further: removing the journal from the reputation transfer.

So rather than a reviewer lending their reputation to a journal, and that journal then conferring a stamp of trustworthiness onto an academic work, reviewers lend their reputation directly to the works they review. The works themselves can still be shared far and wide, e.g. via ArXiv.

Of course, inertia is still a massive force to work against.

[1] https://plaudit.pub

[jonathanstrange]

This is looks like a horrible idea. The system violates anonymous peer reviewing standards. These are sometimes even mandated by law, e.g. our funding institution would not allow us to count such a publication as "peer reviewed." There are many good reasons why peer review needs to be anonymous on both ends (the reviewer is anonymous and the manuscript is anonymized). Many reputable journals have even switched to triple blind peer review, i.e., the editor in chief does not know which two peer reviewers are selected.

The biggest problem I see with the proposed system is that it's unfortunately often easy to recognize who wrote a paper (which is bad for ordinary peer review already) and in a personal endorsement system this would lead to collusion among researchers with low integrity. You wink through my papers, I wink through your papers. Also, imagine you don't endorse the paper of the senior researcher in charge of your postdoc funding...

[Vinnl]

In practice (as you partly allude to), the problems you speak to already often occur for traditional peer review, except they're less traceable there. Sure, collusion rings and nepotism is (still!) possible, but the data is out there for everyone to see and to call you out on. Now winking through a paper risks your reputation (the very thing that makes your endorsement relevant!), rather than only being potentially noticed by an observant editor.

(Something similar goes for researchers in charge of your postdoc funding: how many co-authorships are earned, and how many are ways to game the current system?)

I'm certainly not saying that a public endorsement system is the end-all-be-all and won't have its own problems. However, I do get frustrated every now and then by the institutional inertia that arises from holding new initiatives to higher standards than existing ones (see also: using the Impact Factor to evaluate academics).

[jonathanstrange]

I don't think it's a good idea to explain away valid criticisms as inertia. I also don't buy the implicit claim that giving up the anonymity of the peer reviewers could improve peer reviewing. It seems obvious that giving up anonymous peer review would make things worse and that putting the reviewer's reputation on the line would not improve things. It creates all kinds of conflicts of interests and biases. You also need to understand that the people who game the system have a low reputation anyway. They find a niche, build up their collusion ring with people they meet at conferences, and get a tenure-track position or full tenure because of their high, though low quality publication track record. Moreover, if the peer review is properly anonymous, then reviewers are way more critical than if it's not anonymous. Hence, acceptance rates in top journals in my area are below 5%. If the reviewer is not anonymous, they would be way more permissive, since they don't want to completely shatter their relations with their colleagues. Higher acceptance rates means lower value publications, not higher standards.

There are many things that needs to be addressed to improve science, not just peer review. Hiring policies need to change and indicator counts need to be used more adequately, empirical studies need to be pre-registered and the p-value needs to be 1% or lower. Peer review is just one factor and mostly a monetary issue. If all research institutions would spend enough money to make all commercial journals available for everyone, there would be no particular problem with peer review. The problem of many researchers in poorer countries is that they don't. The problem is not that peer review does not work.

[sitkack]

Anonymity removes the difficult aspects of human social interaction and allows everyone to focus on the science.

Peer review itself is wonderful and is what enables science itself to work as well as it does.

[deepburner]

Frankly "peer review" is the furthest thing away from wonderful. While the idea is legitimate, it is undermined at every step of the way.

* Reviewers are basically forced to it for free. If you're a researcher you won't have the time to properly fact check a 40 page paper for free, it just isn't going to happen. So they (we) don't. The review process isn't very rigorous.

* Journals are leeching off of the public money by double whammying both the scientists and the institutes while providing minuscule value. Their gains are absolutely disproportionate to what they provide.

* Even the most involved and convoluted peer review processes aren't as anonymous as you think. When there's already a handful of people in your field, you can pretty easily tell someone's identity from their writing style.

I would love to pretend that the system is great and works very well but it just doesn't and this pretense and the phenomenon of holding new systems to a higher standard than the current one is preventing us from actually improving the system.

[Vinnl]

I don't mean so say that criticisms aren't valid; just that perfect is frustratingly often the enemy of better.

For example, the status quo can be justified by "it seems obvious that" rather than being evidence-based. In practice, single or double blind is very often not actually blind [1], in which case it's hard to argue that it would be any different than transparency. Likewise, a new solution "creates all kinds of conflicts of interests and biases", without even considering whether the existing CoIs and biases are any better. Likewise, if people who game the system today have a low reputation, why would the same not hold true in a new system? Likewise, does anonymous peer review actually improve the quality of the reviews (e.g. [2])?

It's good not to blindly embrace something new, but I think it's important to withold judgement too, and see how it plays out in practice, and to make an effort to compensate for the prejudice people typically have towards the status quo.

And yes, absolutely agreed that much more needs to be addressed to improve science. But I also very much take issue with the idea that peer review is fine as-is.

[1] https://absolutelymaybe.plos.org/2017/10/31/the-fractured-lo...

[2] Heading "quality of feedback": https://plos.org/resource/open-peer-review/

[jonathanstrange]

The point is that there is nothing inherently wrong with blind peer review, you're proposing the classical solution to a nonexistent problem. The problem is that too many reputable journals were lured over to big publishing houses and these want to use them as a perpetual money-printing machine. To fix the problem, you create free, voluntary-based reputable open access journals with triple blind peer review that will (slowly) replace the expensive ones. This is happening.

Maybe you should consider that you're the stubborn one who links to blog posts from open access journals with a strong bias to reform the peer review system as evidence. Plos journals have this agenda because they don't have good enough pools of reviewers, which is the problem of all new journals. However, it's not as if scientists all over the world and funding authorities haven't thought about the topic. People should not withold judgement about these issues, that's an odd request and would weaken the position of scientific staff towards political decision makers who certainly won't withold judgement.

To give you an example of what is being done by funding authorities, at our university we are required by law to publish every article in openly accessible format. Otherwise they do not count at all towards our salaries, they will be ignored completely. There are public open repositories for that. The only thing that bothers everyone is that big publishers like Elsevier only allow draft versions in those repositories and there is sometimes a 1 year mandatory delay until they can be put online. That needs to change and EU authorities are working on it.

[hoseja]

I'm sure if you applied blockchain and crypto enough times, this could be solved in some byzantine and massively inefficient way.

[jonathanstrange]

Don't forget to make scientists constantly mine for cryptocoins on their GPUs if they want to keep their crappy postdoc job, that way it's profit for everyone. :P