[low_tech_love]

“…because conferences have made it mandatory to review 4-5 papers if you submit to them.”

Is that really a thing? So if I submit a legit paper it is being reviewed forcefully by a bunch of random people from god knows where?

[emil-lp]

That's correct. And 4–5 is low-balling.

I was forced to review 7 papers.

Now, how are these papers chosen for you?

You get to bid on which to review.

Bid on 30 papers out of 30,000.

I got none of those, and I had to review 7 papers I wasn't really competent enough to review.

[YeGoblynQueenne]

For me at that point it's OK to give the paper a quick read, write a few words on what it's about and whether the paper looks well written and so on, explain it's outside my area of expertise and give a weak accept with low confidence.

So basically leave it up to the Area Chair. It still helps them a bit that you give the paper a quick once over and flag it up if it's somehow complete crap. The problem is that I'm probably missing the chance to review papers I'd be really interested in and that I could help to improve.

I guess it's a bit ridiculous that AI conferences in particular can't find a systematic way to route papers to subject matter experts. I guess the usual system with area and sub-area keywords is overwhelmed by the extreme increase in numbers. This year AAAI made link to my Google Scholar and DBLP, link my top five papers, and add some more info about my subject to my profile; and they still failed to rate the papers that were really relevant as most relevant. At least the bidding pool was a bit smaller this time so I guess they're trying to work something out.

Meanwhile submission numbers are exploding and I fear all the conferences can do is try to play catch up.

[tgv]

30k papers? What field is that? How do you even select your area of competence from such a haystack?

I went to smaller conferences, I suppose, but back then I also had to review papers outside my direct expertise, although nothing too remote. But I didn't know the literature well, obviously. I was able to weed out the sub-par papers (I think), but it was harder to estimate the merits of those that did make sense, as I only saw 1 or 2 per area. And that's what determines the acceptance, after all. No criticism because the reviewer judged it perfect or because the reviewer didn't know what to look for? Outcome is the same.

[emil-lp]

AAAI 2026 had a record breaking 27k submissions.

It doesn't scale.

[probably_wrong]

Speaking for the *ACL conferences, they are not "random people" in the sense that they must have "(a) at least two papers in main ACL events or Findings, plus (b) at least one more paper in the ACL Anthology or a major ML/AI venue" [1]. So they are published authors.

As for it "being a thing": sadly yes. The number of papers has been steadily increasing over the years and the number of volunteer authors is simply not enough.

[1] https://aclrollingreview.org/incentives2025