[olddustytrail]

> That’s because we’ve basically reinterpreted what “peer review” is.

Who is "we" in this scenario? Because that's certainly not how I've seen peer review work.

The editor would ask a small group of people in the field to act as reviewers and then send them the papers. They review it and send it back with any requests for changes prior to publication.

So they're the peers that are reviewing, not the editor.

[godelski]

Look at the history of Peer Review. What you see post 1950 is pretty different than what you see prior to that. I think this quote is the best one-liner, though I think everyone should dig much more into the question

  > in the early 20th century, "the burden of proof was generally on the opponents rather than the proponents of new ideas.

That is, the reviewers had a higher burden than the authors. The bias is towards acceptance rather than rejection. In a perfect world we could only accept good papers and could reject bad papers, but we don't live in that world. So the question is "when we fail, which way do we want to fail?" Obviously, I'm on the side of Blackstone here

https://en.wikipedia.org/wiki/Scholarly_peer_review#History

[olddustytrail]

But that's the opposite of what the person I'm replying to said. They're saying everything is acceptable and I'm saying it's actually reviewed.

Ok, maybe that's not what you meant. Peer review doesn't reject papers because they don't agree with the orthodoxy; they reject them because they're not competent. Is that what you were getting at?

[godelski]

I'm a bit confused. What you described is what happens today. Yes. That has been my experience too, serving as a reviewer. I understood xondono to be referencing the history that I mentioned, which is where these reviewers didn't exist. So the requirement was different, which is what I'm saying about the burden of proof.

  > Peer review doesn't reject papers because they don't agree with the orthodoxy; they reject them because they're not competent.

This is absolutely false and I don't know a single academic who hasn't seen competent papers get rejected.

Reviewers can reject for any reason. The system is built on trust, but incentivized to reject. "Not competent" is too vague of a term, just like "not novel."

In my other comment[0] I even reference one of the famous works that got rejected 3 times for being "not competent". This isn't a one-off case here, it is a common occurrence. On several occasions I've had to champion papers which were clearly competent yet my fellow reviewers simply were not familiar with the domain (they admitted this during discussion). I've also killed papers for similar reasons (a very rare event as I strongly bias towards accepting).

So I'm sorry, saying papers are only rejected because they are "not competent" is incredibly naive.

And I'm sorry, but the claim that "works aren't rejected because they don't agree with orthodoxy" is simply laughable. There's a long history of peers rejecting discoveries that upset the norms. This has happened to the majority of well known scientists. I'm not talking about like the Church going after Galileo, I'm talking about things like Galilelo arguing with Tycho Brahe or Christoph Scheiner. Einstein was critical of Bohr. Hertz was critical of Bell. The list goes on and on. The criticism was explicitly about running counter to orthodoxy. This is such a common thread in history that there's even Max Planck stated "Science advances one funeral at a time."

[0] https://news.ycombinator.com/item?id=44587535

[542354234235]

But wouldn’t a consequence of failing to err on the side of caution and “orthodoxy” mean a proliferation of junk and pseudoscience? We already had that problem now but the change you are proposing seems to put a foot on the gas for those issues.

Sure, some valid papers get rejected, but how many bad papers are also rejected? How would it affect the wider scientific community if 1 additional good paper is “approved” and also 100 bad papers? Does good science still make it through eventually, or are these rejected papers losing valuable insights “forever”? What kind of damage can a bad paper cause and how often?

It just seems like it isn’t as clear cut as “the current system is flawed…and the alternative is objectively better” to me.

[godelski]

I think it is good to have that concern, I just don't see strong evidence that more open publishing results in more junk. I hear the arguments that there's already a lot and erring that way would only create more, but tbh, I think it would be mostly the same. Frankly, reviewing just doesn't scale very well. Even plagiarized material routinely gets published in high ranking venues. I just don't see how this gatekeeping provides strong protections against that, but I do see how it actually perpetuates it.

I know that last part sounds weird, so let me explain with an example[0]. This paper was rejected for plagiarism. What's unique about this case is it is public. ICLR lays it all out. You can click on the names and look at their g-scholar pages or better, DBLP, which allows you to better look at who they co-author with. Maoguo Gong here has almost 30k citations and Qiguang Miao just over 10k. The problem is that were they to publish elsewhere, we wouldn't have this record. So it makes it hard to track. Sure, we can open up rejections in all venues, but at that point we're honestly not too different from what I (and others) am advocating for.

What we're advocating for is something like people directly publishing to OpenReview, where we can have the comments on record, link to GitHub, datasets, or whatever. I'd even go further and allow different formats of publication, like blog style. PDFs are great for some domains but not for others. 8 page limits are great for some topics, but also not for others.

IME, there are very few (by percentage) bad actors in the space. That's much different than the percentage of bad papers, mind you. But there are larger problems that incentivize bad papers. Rushing for deadlines and the publish or perish paradigm are the two most obvious.

Importantly, I think we also need to ask who papers are being written for? IMO, it is incorrect to write them for broader audiences. That is the word of science communicators. Where that line is, I think is very subjective. I'd rather the authors decide who their audience is.

I'm all for better communication, but I've only seen the current system make communication worse, not better. Again, this sounds counterintuitive, so I want to provide another example[1]. This is an egregious case where I think the problem is clear to a wider audience (I don't know your background), but frankly, I see stuff like this happen all the time. Where something simple is convoluted to make it appear more rigorous. One of the reasons I believe this happens is due to the need to get past the gatekeepers. Truth is that most reviewers do not spend much time with most papers. To really understand a paper you tend to need to spend hours with it (it's a lot of work!). Frankly, reviewers like to reject because it makes the job easier[2], and the system incentivizes this. The obscurification doesn't exactly come with malintent, but rather small steps. It can even be something small like adding an unnecessary math equation because a reviewer only glancing through will see math and think it better. Unfortunately, we have to play these mind games while writing papers. Yes, better graphics can help communicate works, but we also don't want to stray so far as to make it a priority. Listen to most researchers about how they review papers (which I tend to differ from). They place a lot of focus on graphics and tables. These are important, but they are also meaningless without context.

The problem is that the system is noisy. What people like me are proposing is that we accept and embrace noise rather than sweep it under the rug. The truth is that the noise cannot be avoided. In science, we specify error in our measurements for the same reason. Error is a measurement of uncertainty. By rejecting uncertainty, you only make your measurements less accurate, not more. That's the root of the issue here.

  > and the alternative is objectively better

You're right. And I'm happy to admit that my answer is not globally optimal[3,4]. But I think there is no globally optimal solution. Which is okay. There's many problems with no globally optimal solution. There's always some trade-off. I just believe that the bias should be akin to Blackstone's Ratio rather than the inverse. I believe biasing towards the inverse only makes false positives more detrimental, and we're not doing a great job at tackling the problem. Usually, that means we need to have a substantial rethinking. If the conventional solutions aren't solving the problems, maybe it is time to explore unconventional ones.

Most importantly, I think we need to have an open conversation. I'm certain there are better solutions than the one I've proposed (there's ones I even believe, but they take more to explain and build from here). There's so much we haven't even begun to discuss. It's a hard and complex problem, but isn't that what we researchers are trained for?

But the biggest problem right now is that the conversation tends to be shut down with an appeal to tradition. "Don't fix what isn't broken" is a fine policy, but it only goes so far. Unfortunately, the logic tends to more often be used as an excuse to ignore problems. If there are cracks in the system, surely you want to fix it before it breaks, right? I know I'm not alone in believing that the damn looks ready to burst. I'd like to try to avoid that fallout, if possible.

[0] https://openreview.net/forum?id=cIKQp84vqN

[1] https://youtu.be/Pl8BET_K1mc?t=2456

[2] Personally, I don't. I still spend hours with bad papers and will write lengthy reviews. I'm "on their team". I want them to make the best work that they can. The only "easy" reviews are really good papers and really bad papers, the former being exceptionally rare. But this is also why the review process becomes so subjective. It's very hard to define what constitutes very good and very bad. But I don't think I'm "doing my job" if I am just looking to be done with the job. I am "doing my job" by reading the work earnestly and providing the best feedback I can. I'm not going to waste my time, I'm not going to waste the authors' time, and I'm not going to waste the time of the next reviewer down the line that reads it when it gets resubmitted. I'll mention that I've been frequently recognized as an exceptional reviewer. Not to brag, but rather to give evidence to the implicit claim that I have expertise here.

[3] https://news.ycombinator.com/item?id=44587677

[4] https://news.ycombinator.com/item?id=44587822

[pessimizer]

I think they meant "reinterpreted" over the last century, not over the span of your personal experience and career.

[olddustytrail]

They're saying it changed to not being reviewed properly and I'm saying from recent experience that it is.