[godelski]

Dog whistles in academic reviewing; phrases that can be valid but are typically being used in an invalid way:

- Not novel: Novelty is ill-defined. Most journals and conferences have guidelines on this that come down to "will someone find it useful" which should make "not novel" a rare review, not the most common. Additionally, what's obvious post hoc is not obvious a priori (I actually saw an AC override reviewers because of this!). If something is well written it also comes across as easy and simple. Lack of novelty could be a true lack of novelty or just an indication of a well written paper.

- Incremental: All research is incremental. Research can be too incremental (not enough work done) but this is never expanded upon in those types of reviews. This kind of statement is meaningless in isolation. Incrementalism will always exist with a publish or perish paradigm. One cannot publish breakthroughs year over year. Those take significant periods of time or a lucky break. Neither is a common occurrence, by definition.

- Not enough experiments: "Money is all you need" has become a popular phrase. There may not be enough experiments and this can be true but perfection is the enemy of good. Especially in research, where the experiment space is non-exhaustive. Can a reviewer be satisfied in a finite amount of time and with finite resources?

I've started to change my mind about where blame falls for all this. It starts with the reviewers, but if this becomes the norm then the system is at fault. It has not removed the rotten apples and thus let the barrel spoil. All while we've had decades of discussion about this happening. I think a lot falls on the Area Chairs and Metareviewers, as they should be preventing these types of reviews to pass. But the whole incentive structure of publishing is wrong. There is high pressure to reject and zero pressure to let papers through. This especially hurts our junior researchers (grad students) as it becomes a lottery process for determining if they can graduate. After all, we can't have any wizards without any noobs.

[ThereIsNoWorry]

It is absolutely a systemic problem. A problem that will not be changed or solved within our lifetimes, that's for sure.

[godelski]

> A problem that will not be changed or solved within our lifetimes

Honestly, I don't like opinions like this. That's just passing off the work that needs to be done to fix these problems. It wasn't even a hundred years ago when people published more in the open and the publish or perish paradigm didn't exist. The latter is a relatively tool, and collapsed because Goodhart's Law.

We can make the changes, but not with defeatist attitudes.

[ThereIsNoWorry]

Fair point. I was just thinking that one sensible change could be to publish everything without significant constraints on open platforms. Then, in some way, the community and some moderate moderation would basically decide which papers and researchers get "famous". In the best case, that would result in more true "breakthroughs", but reflecting about how social media works (incentives to constantly push narratives, hype and simple "truths" that explain complex world issues) at the moment, that seems a bit too optimistic. That's why I peddled back to "not solvable within our lifetimes". Who knows. I'd like to do research, but if I go into academia I'd be immensely constrained and would have no freedom what to work on. The only viable option is to become sufficiently wealthy to fund myself and my ideas. Would be nice, even if a tiny bit unrealistic.

[godelski]

There's a double edged sword here. Currently CVPR has a social media ban that is extremely strict. I've definitely been seeing Twitter act differently than it normally does around this so we'll see if it works. But why this matters here is that there's an incredibly strong correlation between the popularity of the lab, the amount of eyes that can read the preprint (with authors names), and acceptance at top conferences (i.e. CVPR). I often say that double blind only exists for small fries. But even still, big labs will get more likes from twitter accounts that post arxiv papers. There are plenty of means to bypass this social media ban.

But even from this experiment we can see that there are people actively trying the fix the situation. Likely naive, but something is better than nothing. I wish there would be more push for ACs and Meta Reviewers having the highest standards, but this is where we are.

I am totally for us abandoning journals and conferences and just publishing to arxiv and open review. This is exactly what I would do if I was independently wealthy and could research without the constraints that I am in. But we also need to recognize the issues with this and that popularity and outreach ability greatly affect perceptions of quality of the paper. That this is not meritocratic.