[godelski]

  > Reviewers know what venues they are reviewing for, and attentive ones will adapt their review based on the prestige of the venue.

Works that way in theory but I've seen people be stricter in an ICML workshop than CVPR.

I don't think it's constable that luck plays a big role. Do we need you do a third NeruIPS study to convince people?

The real problem is that we don't actually know if an idea is good or not until it's had more time to be explored and studied. A great example of this is diffusion models. There's 6 years between Sohl-Dickstein's paper and Jonathan Ho's. All because GANs got popular, so only a few people kept looking at diffusion until one person scaled it. There's hundreds of cases like that, including attention and resnets (I'll defend Schmidhuber's Highway Nets here). So much fruitful research gets cast away for no good reason.

A reviewer can't ever determine if research is good or impactful. It's impossible to do by just reading a paper. So that needs to be taken out of the equation. What a reviewer can do, though, is determine if a paper is bad or fraudulent. So IMO, we should publish anything that isn't fraudulent. Let time tell us the impact, because history tells us we're not very good at figuring that out ourselves