[sawwit]

> I'm somewhat skeptical of how much the model in which "the scientists" do all the work of publishing a journal can scale, but I'd be glad to be proven wrong.

I'm quite optimistic. There are already strong explicit indicators for the worth of a publication. If a paper comes from a university, for example, you can likely already be assured that it's worth reading. If it has a name with good reputation on it, that's an even better piece of evidence. If a paper has interesting content, it will be shared in the community. Also, Google isn't just ranking by a explicit measures, but mostly by PageRank, i.e. a measure how how well a certain item is woven into the network of links (plus possibly hundreds of heuristics). PageRank could likely be applied to a publication system as well. In that case the network links could be co-authorships, associations with accredited universities, and perhaps other things. And let us not forget that the vast majority of researchers are actually truth seeking and concerned about the impact they make on the world. Things might become a little bit noisier, but at the same time the feedback-loops become shorter, as we're seeing it in the machine learning field.

[baghira]

exgrv's commment, the way I read it, was arguing for something less radical: "simply" insourcing the work of the journal publishers, and having it done by scientists (maybe allocating more resources to scientists so that the workload remains sane).

What you are proposing (arxiv.org + PageRank) is quite a shakeup. My impression is that while pretty good at using the position in the graph to establish relevance, such a model is much less effective at gauging quality (which is the problem, if you want to use bibliometric scores as a way to establish whose careers are to advance). In other term, the outcome of a search for "cloud computing" is certainly pertinent with the subject. That is not how you would choose which cloud service to use.

Of course, it may well be possible that the human judgment component is codifiable in a few hundred/thousand of heustics (sounds like a hard problem, but it's not my field), thus allowing the construction of a good model.

[sawwit]

Intuitively I think, the problem is much simpler compared to web search because it's a much smaller graph and each node gets checked against reality in some sense, while on the market and on Google things mostly only get compared with the competition under very obfuscated circumstances with extremely weak feedback loops. PageRank is probably unnecessary for most areas as they are small enough so that specialists can easily keep track of new publications. It could just be a useful tool for listing a lot of publications, but, as I said, measures like citation, review count and reputation of the institution are probably pretty good on their own. It was perhaps misleading that I've mentioned PageRank at all, it was just an idea that I had at the time I wrote the comment.