[hospadar]

Papers are absolutely judged on impact - it's not as though any paper submitted to Nature gets published as long as it gets through peer review. Most journals (especially high-impact for-profit journals) have editors that are selecting interesting and important papers. I think it's probably a good idea to separate those two jobs ("is this work rigorous and clearly documented") vs ("should this be included in the fall 2023 issue").

That's (probably) good for getting the most important papers to the top, but it also strongly disincentivizes whole categories (often very important paper). Two obvious categories are replication studies and negative results. "I tried it too and it worked for me" "I tried it too and it didn't work" "I tried this cool thing and it had absolutely no effect on how lasers work" could be the result of tons of very hard work and could have really important implications, but you're not likely to make a big splash in high-impact journals with work like that. A well-written negative result can prevent lots of other folks from wasting their own time (and you already spent your time on it so might as well write it up).

The pressure for impactful work also probably contributes to folks juicing the stats or faking results to make their results more exciting (other things certainly contribute to this too like funding and tenure structures). I don't think "don't care about impact" is a solution to the problem because obviously we want the papers that make cool new stuff.

[godelski]

> Papers are absolutely judged on impact

This is post hoc thinking but impossible a priori. You're also discounting the bias of top venues, in that the act of being in their venue is a causal variable for higher impact if you measure by citation counts.

I'd also mention that ML does not typically use a journal system but rather conferences. A major difference is that conferences are not rolling submissions and there is only one rebuttal available to authors. Usually this is limited to a single page that includes citations. You can probably imagine that it's difficult to do an adequate rebuttal to 3-4 reviewers under the best of circumstances. It's like trying to hold a debate where the defending side must respond to any question from the opposition, with clear citations, in a short time frame, and there is no limit to how abstract the opposing side's question need be. Nor that their is congruence within the opposition. It's not a very good framework for making "arguments" more clear or convincing, especially when you consider that the game is zero sum.

I definitely agree with your comments about how other types of useful communication (like null results) are highly discouraged. But I wanted to note that there's a poor framework for even "standard" works.