[godelski]

As another ML researcher I'll second this. When I review papers for conferences it takes me hours to review a work that's in my niche. It's because you can't skim a paper unless it is exceptionally good (lol) or has major flaws. A single sentence often holds the magic keys to making an algorithm work, and I don't think many would realize this unless they try to reproduce works from the paper alone (not using lucidrains or the official implementation). Even lucidrains makes some of these mistakes. And yes, even in my own niche, I go back and reread papers that are key references to make sure I didn't forget key details and understand the exact limitations the authors are addressing. The main thing is that the closer it is to my exact same niche the fewer reference papers I have to read (because I know which ones matter) and the faster I can read those papers. It's ensuring I am not forgetting key nuances of specific datasets or specific metrics that are used, because not keeping these in mind will trick me into wrong conclusions. This is what's required if I want to give a quality review. It's what's required if I see myself as on the same team as authors (team science) and helping them make the best work they can.

But I'll admit that there's a lot of pressure for me to stop doing this. A big part being that it's very clear my reviewers are not prescribing to this tactic. Rather I think many reviewers are not concerned with the rigor of their reviews. That they do not see themselves on the same team but rather antagonistic (team conference/team journal) and that their job is to filter. But I think an issue is that in ML you get an advantage if you are reject heavy and lazy in reviewing. Not only do you save on the time it takes to review but since it is a zero sum game you ever so slightly increase the odds of your own work being accepted. Honestly I do not feel the process is very scientific. Even the new CVPR LLM rules are a joke. More signaling than solutions. I just wonder if people care about the science anymore.