[godelski]

> Papers are absolutely judged on impact

This is post hoc thinking but impossible a priori. You're also discounting the bias of top venues, in that the act of being in their venue is a causal variable for higher impact if you measure by citation counts.

I'd also mention that ML does not typically use a journal system but rather conferences. A major difference is that conferences are not rolling submissions and there is only one rebuttal available to authors. Usually this is limited to a single page that includes citations. You can probably imagine that it's difficult to do an adequate rebuttal to 3-4 reviewers under the best of circumstances. It's like trying to hold a debate where the defending side must respond to any question from the opposition, with clear citations, in a short time frame, and there is no limit to how abstract the opposing side's question need be. Nor that their is congruence within the opposition. It's not a very good framework for making "arguments" more clear or convincing, especially when you consider that the game is zero sum.

I definitely agree with your comments about how other types of useful communication (like null results) are highly discouraged. But I wanted to note that there's a poor framework for even "standard" works.