|
Hi! I'm the research lead for Anthropic's interpretability team, and was the decision maker for us publishing our papers web first and not doing traditional publications. A few thoughts: (1) As others have commented, I think peer review in ML is pretty widely accepted to be dysfunctional right now. I think most people who have published in ML conferences would agree. It's not unusual for early PhD students and sometimes even undergrads to review, and reviewers are overburdened to the point where they can carefully consider all their papers. Everything I've said so far is just anecdote and opinion though. A more objective test was the NeurIPS 2021 Consistency Experiment ( https://blog.neurips.cc/2021/12/08/the-neurips-2021-consiste... ) which found that if a paper was accepted by the conference, there was only a ~50% chance that a parallel review process would come to the same conclusion. (2) Modern peer review is a relatively modern invention, arising in post-WW2 science as the scientific community grew dramatically, and there was a need for more systematized ways to make decisions about publication, funding, jobs, etc. Famously, Einstein was offended by one of his papers being sent for review. I don't think it's at all obvious that this transition has been good for science! I see lots of people writing papers for reviewers, rather than with the goal of doing the most impactful science they can. (3) As background, I spent 5 years of my life running a scientific journal ( https://distill.pub/ ), trying to have excellent review processes and enable non-traditional papers to be peer reviewed. I honestly just burnt out on this. Now I just want to do good research. (4) We do circulate draft papers to researchers working on similar topics at other industry groups, and in academia. As other comments have noted, this sometimes leads to public comments on our papers. In many cases, these are a much deeper review than you'd see in typical peer review processes, such as independent reproduction of experiments. |