[czr80]

So many of these discussions take it for granted that publishers don't do anything of value. This blog post, by an open access advocate, was for me very illuminating on what value they actually add (http://cameronneylon.net/blog/polecon-of-oa-publishing-i-wha...)

Quoting from that link: "One of the big challenges is discussing the costs and value added in managing peer review is that researchers who engage in this conversation tend to be amongst the best editors and referees. Professional publishers on the other hand tend to focus on the (relatively small number of) contributors, who are, not to put too fine a point on it, awful. Good academic editors tend to select good referees who do good work, and when they encounter a bad referee they discount it and move on. Professional staff spend the majority of their time dealing with editors who have gone AWOL, referees who are late in responding, or who turn out to be inappropriate either in what they have written or their conflicts of interest, or increasingly who don’t even exist! .... Much of the irritation you see from publishers when talking about why managing peer review is more than “sending a few emails” relates to this gap in perception. The irony is that the problems are largely invisible to the broader community because publishers keep them under wraps, hidden away so that they don’t bother the community."

I'm sure there must be some way to achieve this and also make the content freely available. But whatever system replaces the current one has to deal with these issues too.

[return0]

> I'm sure there must be some way to achieve this and also make the content freely available. But whatever system replaces the current one has to deal with these issues too.

Post-publication peer review could solve all of these (and the access issue too!). I'm still not convinced of the superiority of pre-publication review by only 3 (usually quite busy) people max.

[dalke]

I haven't had good experience with modern post-publication peer review so far.

My biggest complaint is that few know those comments exist. For example, I commented on one paper to highlight methodological problems that make it impossible to trust their results. (They developed a new algorithm for X. They compared it to the naive implementation. The naive implementation was poorly coded. Most of the speedup disappears by in comparison to a well-written implementation.)

I've talked to a few people about this exact paper. None noticed the link to the comment page.

I say "modern" because post-publication peer review isn't new. One of the older mechanisms was the letter to the editor. Those letters (at least in the ACS journal I'm thinking of) have a DOI and are searchable. A few of these letters have proved useful to my research.

But the modern post-publication peer reviews don't have a DOI and aren't indexed by Google Scholar or other systems, so they are less useful than a old Letter to the Editor.

(I once asked about sending a Letter to the Editor to an Open Access journal, concerning problems in one of their publications. They said they don't support those sorts of short communications, and my only option was the $1,500 to publish a full paper, which would have to go through the normal peer review process.)

[Fomite]

I have been extremely unimpressed with my experiences post-publication peer-review. Limited to no feedback for many papers, and some clearly "How dare you stray onto my turf" related comments.

[visarga]

You mean Elsevier doesn't need to earn $2 billion for hustling free editors to do their work proper? You know, it's more than just pushing emails around. You have to deal with people.

[return0]

Have you accounted the cost of the many work-hours that reviewers spend beautifying their journals? it's probably in the tens of billions, worldwide. Is it justifiable? It's certainly not what the public is paying for.

[maaku]

I think you missed an Office Space reference.

[KKKKkkkk1]

I don't understand. What is the difference between post-publication peer review and no peer review? What is a journal for if the papers are not peer reviewed?

[kriro]

The simplest workflow I can think of:

1) Publish with no review

2) Open for review

3) If review is passed, tag as reviewed and bundle all those into a special reviewed section (which can be cited)

4) Reserve the right to remove from reviewed (some later reviewer finds major flaws etc.)

You'd have to handle the dynamic nature of papers (git-like) but in essence I'm thinking of a website with "all submitted papers" and a checkbox for "reviewed papers only". I think that would be valuable and people would likely leave the checkbox on by default.

I'm also curious if the review has to be blind in this model. I think it would be more valuable if the reviewers actually signed with their names and the review feedback was open as well. Shifts the value from being in a prestigious journal to being successfully reviewed by a smart/reputable person. Ideally doing these reviews would then become reputation building as well.

[return0]

It's not a journal, or it's a different kind of journal. I guess it would be like doing the whole review process in a very public way. Would that be worse? I don't know ... has anyone tried it?

I think we are heading that way anyway, the open access publishers are pushing the envelope here by making peer review more and more public (publishing reviewer names, or comments along with the paper).

[notalaser]

Just saying -- perhaps "editors who have gone AWOL, referees who are late in responding, or who turn out to be inappropriate" would likely be less of a problem if they were properly paid for what they're doing. I haven't been in academia for a long time (and my stint was short), but I don't remember ever hearing of a reviewer who got anything more substantial than maybe a discount or a one month of free subscription to a journal from a publisher. Managing peer review is admittedly more than "sending a few e-mails", but not in a good way.

[dalke]

What is "properly paid"?

The last review I did (for an open access journal) took ~6 hours. At minimum wage that's about $50. There were two reviewers = $100. Reviewer must get paid even if the recommendation is "don't publish", otherwise reviewers will have an incentive to say "yes". The OA fee for that journal is about $1300. If 50% of the peer reviewed papers are published, then the average price for peer review would be $200, which would raise the price to publish by about 10%.

It is an open access journal, so everyone effectively has a free subscription to it.

One alternative is a credit model, where N reviews gives a discount on the price to publish. However, as the above points out, it may be more cost effective to mow lawns on the weekend than it is to plan for that discount.

Then there's the question of perceived fairness. If one reviewer says "great job. Publish" and another gives 10 pages of useful critique, is it proper to pay each the same?

[notalaser]

My information may be out of date because, as I've said above, my dance with academia ended a long time ago. AFAIK most (all?) of Elsevier's journals aren't open access and reviewers aren't paid at all. They aren't improperly paid, they're not paid at all. The publisher gets a hefty amount of income out of their unpaid work, of which they share exactly nothing. This is well past the level where we can debate nuances of fairness.

I don't know enough about this field to be able to properly debate the proper mechanics of payment. I just wanted to point out that one of the pillars of these publishers' income -- the credibility brought by peer review, such as it is -- is effectively the unpaid labour of a lot of very, very clever people, who contribute enormously to human progress (not just to the publishers' pockets) pretty much for free (sure, some of them are also assholes, but that's besides the point). I can understand why sometimes they'd be less then collaborative. Over here, in the industry, if I were offered the (industrial) equivalent of that deal, my answer would be a warm and heartfelt fuck you; in fact, I don't think any serious company would want to compromise its image by offering such a deal.

Edit: at the risk of appearing to suffer of the anti-academic sentiment that is so plaguing our profession (which, I have to insist, I do not), I also think that some of this problem is self-inflicted and has to do with the way in which modern society treats higher education. I sometimes think that part of the solution could lay in discouraging the short-sighted, what-can-we-reliably-solve-in-no-longer-than-three-years, paper-focused approach to research activity that is self-feeding the publishing machine.

[dalke]

Referee wrangling is not unique to Elsevier. While you are correct about "one of the pillars of these publishers' income", I have heard very few argue that referee compensation would improve the situation.

To the best of my limited knowledge, the complaints about reviewers taking a long time or not responding at all are independent of the publisher, and equally true for open access journals. I've gotten my share of "the deadline is in two days" emails. No can I think of what compensation might entice me to respond in three days rather than three weeks.

The blanket statement "Over here, in the industry" does not universally apply. In my field of pharmaceutical chemistry, I believe most research papers are from industry, they participate in the free peer review model, and do not believe their image has been compromised.

Your industry may well be different; perhaps it doesn't have a large research component?

[notalaser]

> I have heard very few argue that referee compensation would improve the situation.

Referee compensation alone? No, not at all. I think compensation would help to some degree, but it wouldn't bring an end to these problems.

What I argue is simply the point that managing editors is a very complex matter that is extremely expensive and somehow justifies what publishers are charging for it. It's certainly not as trivial as just sending e-mails (because of the, uh, human factors), but obviously the bulk of it is done by the reviewers -- and they basically do it for free. If this money, or at least part of it, were to go to the reviewers, the claim would at least be credible. As it stands now, it's simply not.

It's as if I were running a paint shop, charged clients a thousand times the sum of the paint (claiming that I do more than just splat paint on cars -- which would be arguably true), but then paid zero wages to all employees except for a few supervisors (not to mention magically reusing every bucket of paint I ever bought, just to keep the analogy correct). Sure, I'd be doing more than just splattering paint on cars -- but a-thousand-times-the-price-of-the-paint-more, when I'd basically have an endless supply of paint that I'm given more or less for free, and only have to pay like 10% of my employees?

Maybe the scientific publishing business isn't profitable enough to allow for proper remuneration (and defining "proper" is also difficult). But, leaving aside the - possibly idealistic - observation that it's probably important enough that maybe it could be worth doing it for no profit at all (or at least for something somewhat more modest that 2 billion dollars!), that's certainly not an argument for keeping it unfair, too. Surely, some payment, even if meager, would at least provide some peace of mind for some of the reviewers, and is arguably better than no payment at all

> Your industry may well be different; perhaps it doesn't have a large research component?

Hm. I guess my claim about industry isn't entirely fair, seeing how the field in which I did academic work (briefly and at a very basic level) is not quite the same as the one I'm active in (tl;dr a niche in microelectronics back then vs. computer engineering now). In any case, indeed, I think neither of these fields have as large a research component as pharmaceutical chemistry (microelectronics as a whole probably does, but what I was doing wasn't as fancy as the name of the field would imply).

[dalke]

> but obviously the bulk of it is done by the reviewers -- and they basically do it for free

Agreed. 6 hours @ $200/hour for my consulting rates gives $1,200. I give that away for free. What remains is the non-bulk. That's still expensive. Who will do the typesetting and proof reading? In physics and math, this is often pushed into the TeX stylesheet. Not so for most other field.

One of the journals I've reviewed for has a box for "does this paper need to be reviewed by a professional statistician?" That costs money.

Journals also check for ethical problems, like plagiarism and attempts to game the system, like http://www.nature.com/news/publishing-the-peer-review-scam-1... .

http://www.michaeleisen.org/blog/?p=1883 recently outlined the costs for PLOS One, an open access biology journal. It costs about $2,000 for them to publish a paper. By comparison, the for-profit journals make about $6,000 per paper. Hence the 40% profit for Elsevier and others. The physics preprint site arXiv costs about $10 per paper, so there's plenty of room, certainly. (There's an increasing growth in "overlay" journals, which build on top of preprint systems.)

If you start to pay reviewers actual money, then accountants get involved. Any idea of what it would cost to manage all that overhead, for residents in countries across the world?

[maaku]

People would be more sympathetic if peer reviewers were actually paid for what they do.