[egjerlow]

Would you elaborate on why you think comments on papers might be dangerous? For me it seems like the most natural step forward, given where we are today.

As I see it, it would solve many issues. Ideally, we would move away from a publication-count metric, and more onto a reputation-based metric. It would lower the bar for participation in scientific discussion, make reproducibility more important, and generally be a healthy thing for science as a whole, I think. But I want to hear opposing viewpoints!

[xioxox]

I don't think comments would be useful. To make a proper comment on a paper requires a significant amount of work (as does refereeing). Lowering the barrier to entry would produce some first impressions, the usual moans about lack of citation and misunderstandings. If a paper deserves a serious comment, I think the proper format for that is another paper.

Although citations are horrible metrics, I think some sort of reputation ratings based on quick likes would be even worse and even easier to game.

[return0]

>To make a proper comment on a paper requires a significant amount of work (as does refereeing).

Let's not fool ourselves, not all refereed papers are great, not all of them are significant and not all of them are published in high impact journals. Most of the reviewing is cursory, menial, and often delegated to postdocs/phds. Most scientific topics are highly specialized, and you 're unlikely to see trolls bothering to comment on them. Writing another paper as response is not a solution either: the pace of article publishing is months and years, not seconds.

An example: I have often found the discussion of scientific papers here, in HN, to be illuminating, clarifying or countering issues, and offering a wider perspective that is often not mentioned in the article itself. And it's not like everyone in HN is a luminary, just mostly inquisitive people. I also see articles discussed in a useful way in twitter. Why can't we open up this discussion?

I don't think comments are to be taken as reputation metrics either. But i do think a simple, helpful open discussion section is missing from every paper that i have read. I believe the main barrier to it is that academics do not want to get off their high horse of untouchability.

To be precise, many journals do have a comment section, but nobody uses it.

[JD557]

>To be precise, many journals do have a comment section, but nobody uses it.

GitXiv[1] also has a comment section and voting features, and nobody uses it.

I know it has a small user base compared to ArXiV, but I think that its format resembles what ArXiv might look like with comments, votes and reproducible code.

[1]: http://gitxiv.com/

[jessriedel]

> GitXiv[1] also has a comment section and voting features, and nobody uses it.

Likewise SciRate is a reasonably popular arXiv overlay, but the comments are never used: https://scirate.com/

[egjerlow]

> If a paper deserves a serious comment, I think the proper format for that is another paper.

That's the thing, though - speaking as a former academic, the reaction when you see a paper that needs serious commenting is mostly "I'll just ignore this paper, since it has these issues", and you continue working on your own stuff - which again is because of the need for citations. If comments had larger effects on your career, they would be worthwhile making. I would guess the effect would be less papers and more 'collaborative science' where you start from a paper and either suggest improvements or improve it / replicate it yourself. I don't see the current proliferation of papers to be a good thing.

As for the worry that there will be too many 'laymen' doing the commenting, there could (as suggested elsewhere) be a verification process - i.e. there could be a 'Verified Ph.D.' (or some other measure of merit) comment section in addition to the 'layman' section.

There could also be a 'replication' section, where replication efforts will be rewarded - possibly by giving the replications some fraction of the paper's total 'reputation'.

In short: If comments are made to really count for scientists, I think there will be a more healthy scientific process going on. Viewing a paper as an evolving thing is IMO more in line with how science does work anyway - a result should be replicated before it's accepted, which is not the case now.

[dalke]

We need only to look to journals to see counter-examples to your proposition that "If a paper deserves a serious comment, I think the proper format for that is another paper."

For example, a journal might have a "Letters to the Editor" section for people to voice a serious comment, even though such a letter is not "another paper".

[Aeolun]

The only thing you have to do to make comments worthwhile is restrict them to actual academics. Even if they went on to post shit, it would only destroy their own reputation.

[adrianratnapala]

This destroys one of the potential benefits of comments -- which is to open up what might be closed communities. Academic disciplines can at times turn into echo chambers.

But there is the rub: if you make comments open, then you will will probably be flooded with. If you close them up, then fail to get the benefits of openness.

[mjohn]

There are plenty of professionals who would be able to contribute, but are not academics affiliated with a university.

[dsqrt]

As somebody in the academia, I would see comments as potentially very dangerous. A comment system could be very easily abused by established researchers in some areas to prevent others from entering their niche sub-field. Do not get me wrong: the majority of scientists I know have very good professional integrity. However, I have also witnessed cases where established professors tried in all possible ways to prevent people from competing with them by discrediting or delaying the publication of very solid science. The arXiv as it is now is a great equalizer, because papers are presented without any string attached and this gives a chance to everyone (or at least to experts) to judge the quality of the various works without pre-conceptions.

As a fictitious example of what could go wrong. I am a theoretical astrophysicist. I am not established (I am not a tenured professor). Suppose that I come up with a good model to explain some astronomical observations. What I would want to do is to write a paper directed to my astronomers colleagues explaining how my model works and why it is better than competing models. I want to convince them to work with me to analyze their data. As it is now, I would post it on the arXiv, the astronomers would probably find it, read it and evaluate it. However, if the arXiv had comments, a single negative review by a more established theoretical astrophysicist would be enough to discourage any astronomer from even reading my paper. Remember that in the astro-ph section of the arXiv there are of the order of 100 new papers per day and we can realistically read only 1 or 2 papers per day on average. In this situation, the chances of my work being completely ignored because of that one comment would be very significant.

I think that the current channels for commenting on scientific work, private email and/or rebuttal papers, are perfectly adequate.

[return0]

"A single negative review by a more established theoretical astrophysicist would be enough" to discredit his opinion completely if your methods and results prove useful to the community. And besides, he can do that anyway if your paper happens to fall in his hands for review, which is a much more vicious system, because all the backstabbing happens in private. I disagree with your premise here.

[dsqrt]

Just immagine having to scan through 100 articles every day looking for new ideas. You find something that sounds interesting from somebody you do not know personally (happens to me all the time). The topic is of your interest, but you are not necessarily an expert of that particular subfield (because you are an observer, a "user" of models, not a "developer"). Judging its quality would take away one day of your work. Clearly, if there is a comment from a famous established professor saying that the work is wrong, you will happily forget about it.

It is also true that a famous professor can backstab you when reviewing your paper. But, first of all, your work is already on the arXiv, so everybody already had a chance to form their own opinion. Secondly, in a peer review process the reviewer cannot arbitrarily reject papers. It does not work like that. He/she has to provide good motivations for his/her recommendations. Reviewers can also be challenged to the editors, who can ask for second or third opinions.

Finally, as others commented, to truly disseminate your work you need to go out and engage the community, giving seminars and talks. I couldn't agree more with this. My issue with that, and I talk as a privileged because I work in one of the top Universities in the USA, is that going to conferences, giving seminars and so on, is way easier if you come from one of the top places. You have funds for traveling, you had a lot of chances to network with the right people (when they visited your institution, for example) and so on. It is much more difficult if you come from lesser known groups or from abroad. The system as it is now already strongly favors people working in the top research Universities in the USA and Canada.

The arXiv is great because it puts everybody at the same level. It ensures that the best ideas have a chance to come out, independently from their origin. I wouldn't want the scientific discussion to be dominated by few loud voices.

[hackuser]

> The arXiv is great because it puts everybody at the same level. It ensures that the best ideas have a chance to come out, independently from their origin.

With arXiv as a basis for comparison, what is your impression of what the other / older methods of dissemination overvalue and undervalue? For example, my impression is that the other methods favor top U.S. research institutions, but maybe that's realistic; maybe the older methods actually undervalue top institutions (despite my egalitarian fantasies). Maybe gender or experience or position or scope or novelty or other things are over/undervalued.

I do notice that in scientific research, institution is almost a surname in people's identities. It's always 'Jane Doe of Harvard'; it seems like it might as well be 'Jane Doe Harvard'.

[dsqrt]

I wouldn't say that traditional journals in my field are biased in favor of top U.S. research institutions. The bias comes from the readers. Someone reading articles that are not directly connected with their research has a hard time judging whether the results are solid or not. It is unfortunately natural that scientist, often without even realizing it consciously, end up using proxies, such as the author's affiliation, when evaluating papers. My worry is that adding comments to the arXiv might institutionalize these biases and contribute to the formation of a more closed and elitist scientific community.

[woodson]

I think his point is that, because of that negative comment his methods and results will never actually put to the test of whether they will be useful to the community; however, given the volume of new papers coming out it is unlikely for that paper to be picked up by the community as is, with or without comments. The best way to get people interested is in establishing direct relationships (ideally in person: go present at a relevant conference, engage with people in the community).

[hackuser]

> I disagree ...

With due respect, do you have experience inside the world of theoretical astrophysics research or something similar? Do you know how it actually works or are you speaking ... theoretically (ahem)?

[Certhas]

It changes the nature of the site.

I fully think open peer review in various forms, is awesome and should be pursued, but the arxiv serves a more fundamental need as well: To simply make the papers available. Within the fields where arxiv is established this is not an issue, but we need to get adjacent fields, and really everyone to agree to a publicly available database of research first. There is still considerable reluctance in some quarters to upload to the arxiv as I found out after changing fields.

The arxiv should be a pure database, then we can think about various ways to build on top. That could be arxiv overlay journals or journals that try out more fancy, new peer reviewing systems that might include curated comments [0].

As long as there is considerable disagreement over what the right format for serious commenting on papers is, arxiv should not be taking sides.

[0] http://www.earth-system-dynamics.net/peer_review/interactive...