This is actually not a bad idea, and some scientific groups are pressuring to move to this model. See Yann LeCun's proposal ( http://www.lecun.com/ex/pamphlets/publishing-models.html ) for an example. ICML almost does this. The review is done privately (due to some well-discussed elsewhere issues with double anonymity), but there's a public discussion site for all papers http://mldiscuss.appspot.com/ . And people do use it, and for some papers you will find important information there.