[jll29]

I haven't seen a manuscript that could not made a better paper through peer review.

Now there are good and bad reviewers, and good and bad reviews. However, because you usually get assigned three reviewers, the chance that there is not at least one good reviewer or at least a good review from a middle to bad reviewer is not that low, which means if you get over the initial "reject" decision disappointment, you can benefit from that written feedback. The main drawback is the loss of time if a rejection means you may lose a whole year (only for conferences, and only if you are not willing to compromise by going to a "lower" conference after rejection by a top one).

I have often tried to fight for a good paper, but if the paper is technically not high quality, even the most original idea usually gets shot down, because top conferences cannot afford to publish immature material for reputational reasons. That's what happened to the original Brin & Page Google/PageRank paper, which was submitted to SIGIR and rejected. They dumped it to the "Journal of ISDN Systems" (may this journal rest in peace, and with it all ISDN hardware), and the rest is history. As the parent says, you want to see people succeed, and you want to give good grades (except in my experience many first year doctoral students are often a bit too harsh with their criticism).

[smcin]

> I haven't seen a manuscript that could not made a better paper through peer review.

Authors don't object to revision suggestions, they object to arbitrary/unfair rejections plus 4-year delays.

> The main drawback is the loss of time if a rejection means you may lose a whole year (only for conferences, and only if you are not willing to compromise by going to a "lower" conference after rejection by a top one).

No, it's much worse if you're a masters student who wants to publish and have that publication accepted within 12-18mths for your jobhunt; you'll necessarily compromise by skipping most journals and aiming for middle-tier conferences (high accept rate and fast turnaround), then falling back on journals only if you get rejected. This was why arxiv was created. (Some academic type here is inevitably going to retort "But conferences (not even those with official proceedings) aren't considered 'real' publications in my field", to which the counter-retort is "the job market doesn't care about that attitude".)

[versteegen]

I haven't seen a manuscript that could not made better, but peer review isn't the best way to improve it.

For a start, even for a journal, the fact that there are multiple reviewers means that none of them feel much responsibility to help improve the paper. They (and I, honestly) oft seem content to read part of it and just provide comments/criticism about that. Especially at conferences the possible verdicts are "reject" or "can't find a reason to reject". I've submitted at least one paper where I wanted the reviewers to point out flaws and reject, and I was left sorely disappointed (it was rejected of course, but not for good reasons) and still in the dark about whether the research had merit.

That said, OpenReview reviews are on average far better than the ones I've received, I think it's fantastic.

[godelski]

I don't think conferences have the capacity to do this. Journals, yeah, conferences no.

The difference is that in a conference you're in a zero sum game and there is no chance for iteration and the framing is opposition of reviewer/author rather than seeing as being on the same team. Yes, every work can be improved, but the process is far too noisy and we can't rely on that iteration happening between conferences.

From personal experience, I've had very few reviews that have meaningful and actionable feedback. Far more frequently I've gotten ones that friends joke that GPT could have done better. My last one I had a strong reject with high confidence by a reviewer's who's only notes were about a "missing" link (we redacted our github link) and a broken citation leading to the appendix. That's it. We reported them, then they got a chance to write a new angry review which seemed to convince the other two borderline reviewers. Most frequently I get "not novel" or "needs more datasets" without references to similar work (or references that are wildly off base) and without explanation as to what datasets they'd like to see and/or why. Most of my reviews are from reviewers reporting 3/5 confidence levels and are essentially always giving weak or borderline scores (always bias towards reject). It is more common for me to see a review that is worse that the example of a bad review in a conference's own guidelines than one that is better.

As a reviewer, I've often defended papers that were more than sufficient and I could tell were making rounds. I had to recently defend a paper for a workshop that was clearly a paper that made a turnaround form the main conference (was 10 pages + appendix when most workshop papers were ~5) and the other two reviewers admitted to not knowing the subject matter but made similar generic statements about "more datasets would make this more convincing." I don't think this is helping anyone. Even now, I've been handed a paper that's not in my domain for god knows why (others in domain). (I do know, it's because there's >13k submissions and not enough reviewers)

I've only seen these problems continue to grow and silly bandaids attempt to be applied. Like the social media ban, which had the direct opposite result of what they were attempting to do and was quite obvious that that would happen. The new CVPR LLM ban is equally silly because it just states that you shouldn't do what was already known as unethical and shifts the responsibility to the authors to prove that an LLM gave the review (which is a tall order). It is like proving to a cop that you've been shot for them only to ask that you identify the caliber of the bullet and the gun that was used. Not an effective strategy to someone bleeding out. It's a laughable solution to the clear underlying problem of low quality reviews. But that won't happen until ACs and metas actually read the reviews too. And that won't happen until we admit that there's too many low quality reviews and no clear mechanism to incentivize high quality ones https://twitter.com/CVPR/status/1722384482261508498