[mycologos]

> the author notes that it is well-known that the quality of the work has little to do with its odds of acceptance even when the system is 'functioning' as designed

I think this is a little more pessimistic than what the piece says. The NeurIPS (then-NIPS) experiment said that about 60% of papers accepted by one PC got rejected by the other. That doesn't actually mean "the quality of the work has little to do with its odds of acceptance". It may just be that there's a paper has to cross a quality threshold, and once it's past, then the outcome has a lot more variation.

My personal take on NeurIPS specifically is that there's a fraction of bad papers, maybe 40%, that probably shouldn't and won't get in. Then there's a minority of very nice papers that probably should and will get in, maybe 5-10%. And then there are a bunch of middling papers where a lot of it is luck and drawing friendly reviewers. But these aren't bad papers, and you can't really just churn them out, they're just not very good papers.

[fulafel]

Seems it's also rational for an opportunistic player of the publications game to split their work into the maximal number of marginally acceptable publications, which would result in a high rejection %.

[xmprt]

I wouldn't be surprised if that's already happening and has become the preferred strategy for the last decade at least.

[Teever]

https://en.wikipedia.org/wiki/Least_publishable_unit

[Beldin]

Don't forget there can be papers recognised universally as acceptable.

From memory, it was 25% rejected by both PCs, 15% accepted by both PCs, and the middle 60% random.

[wiz21c]

then why not review projects in a double blind fashion ? Those which are accepted twice will get published and voilà. But that's twice the work.

[mycologos]

Double-blind usually means something else in reviewing: the reviewer does not know who the author of the paper is, and the author does not learn who the reviewer is. Single-blind reviewing is when the reviewer knows who the author of the paper is, but the author still does not learn who the reviewer is.

As you mention, the problem with having two sets of reviewers is that it's hard enough for conferences like NeurIPS to find one set of qualified reviewers. Usually at least 1/3 of reviewers on any given paper produce a poor review, either because they don't care or because they really lack expertise. Complaining about this is so widespread that it even doubles as a sort of icebreaker for researchers, but nobody has a good solution.