[jules]

Is all of that necessary? Would a model like the following work:

Have a web site where scientists can submit papers. Other scientists can then review the papers. The reviews are publicly (though optionally anonymously) published along with the papers, and the authors of the paper can submit revised versions of the paper.

[CJefferson]

No.

The website would get far too many terrible papers.

Many papers would get no reviews.

Many reviews would be terrible, and done by non-trustworthy people.

The problem with the system you are suggesting is that it feels a lot like wikipedia, and who "wins" on wikipedia has little to do quality, and a lot to do with who is willing to spend the most time editing / wiki-lawyering.

[MikeTaylor]

Sadly, the evidence so far strongly suggests that you're right, and this idealistic approach won't work. That evidence is from PLoS ONE (which remember is the main example of how many innovations do work). They have had the ability for people to comment on and evaluate their papers for a couple of years now, and it's hardly ever used.

[jules]

So can we get the incentives so that scientists do review in such a system?

[jedbrown]

The EGU journals work this way. Official reviewers are assigned, but third parties can comment. An example submission

http://www.the-cryosphere-discuss.net/5/1749/2011/tcd-5-1749...

[alttag]

A hybrid model with both assigned reviewers and public comment is being tried by a regional journal (Southern Association for Information Systems): http://knol.google.com/k/craig-vanslyke/jsais-review-process...

[greeneggs]

This would not work. Reviewers would not feel comfortable being completely honest if they knew their reviews would be published publicly. In any case, there should be no need for the reviews to be public.

[MikeTaylor]

Reviewers should be prepared to stand by their reviews. If they're not prepated for it to be known that they said something in a review, then they should not say that thing. Anonymity and secrecy in reviewing doesn't help the field.

[zbyszek]

Why should the reviewers feel unconfortable? In the current system, reviews are made available to the paper writers, and that doesn't seem to inhibit the reviewers, despite the fact that one often has a good idea of who the reviewers are, especially in the more niche fields of science.

[lutorm]

Even today reviewers run the risk of having their reviews made public, since nothing prevents the author from disseminating it. They are however normally anonymous.