[godelski]

  > because there are just so many papers at leading AI conferences

Ironically a big reason there's so many papers to review is because so many are rejected.

A low acceptance rate is unhealthy, especially in conferences (1 round of review). Papers just get recycled to the next conference, which, as is easy to model, creates an exponential feedback loop. It doesn't explain all the papers submitted, but it sure can explain a lot. Too much rejection is like shooting yourself in the foot.

Not to mention that it's just easy to reject works. All works are flawed, especially works that are in less mature domains. I see plenty a paper get rejected for lack of money. "Not enough experiments" is an common critique that's used inappropriately (along with the highly subjective "not novel enough" one) because it's fine to always want more but no lab has infinite funding. It is used lazily. The question shouldn't be about if your favorite benchmark is used, it should be if there isn't enough evidence to support the hypothesis or not. A mature domain where thousands of people work in it, yeah, that needs stronger evidence. A niche domain where dozens of people work in? Not as many required. Rejecting them ultimately slows down the progress of science because you require any new idea to outperform mature ideas. Ironically killing novelty as no one is going to, or even could (publish or perish), spend all the time and money to mature a niche all on their own.

[nneonneo]

Rejection works when there are multiple tiers of venues; authors often “give up” on a venue if it seems like their work isn’t getting in, which allows higher-tier conferences to maintain a lower accept rate and take only the “best” research. Reviewers know what venues they are reviewing for, and attentive ones will adapt their review based on the prestige of the venue.

Of course, there’s lots of room for subjectivity here; what constitutes the “best” research is still at the whim of reviewers.

[godelski]

  > Reviewers know what venues they are reviewing for, and attentive ones will adapt their review based on the prestige of the venue.

Works that way in theory but I've seen people be stricter in an ICML workshop than CVPR.

I don't think it's constable that luck plays a big role. Do we need you do a third NeruIPS study to convince people?

The real problem is that we don't actually know if an idea is good or not until it's had more time to be explored and studied. A great example of this is diffusion models. There's 6 years between Sohl-Dickstein's paper and Jonathan Ho's. All because GANs got popular, so only a few people kept looking at diffusion until one person scaled it. There's hundreds of cases like that, including attention and resnets (I'll defend Schmidhuber's Highway Nets here). So much fruitful research gets cast away for no good reason.

A reviewer can't ever determine if research is good or impactful. It's impossible to do by just reading a paper. So that needs to be taken out of the equation. What a reviewer can do, though, is determine if a paper is bad or fraudulent. So IMO, we should publish anything that isn't fraudulent. Let time tell us the impact, because history tells us we're not very good at figuring that out ourselves