[DrScientist]

The current system lies on the market of ideas - ie if you publish rubbish a competitor lab will call you out. ie it's not the same as the two people in an aircraft cabin - in the research world that plane crashing is all part of the market adjustment - weeding out bad pilots/academics.

However it doesn't work all the time for the same reasons that markets don't work all the time - the tendency for people to choose to create cosy cartels to avoid that harsh competition.

In academia this is created around grants either directly ( are you inside the circle? ) or indirectly - the idea obviously won't work as the 'true' cause is X.

Not sure you can fully avoid this - but I'm sure their might be ways to improve it around the edges.

[bjourne]

> The current system lies on the market of ideas - ie if you publish rubbish a competitor lab will call you out.

Does not happen in practice. Unless you're driven by spite, fanaticism towards rigorousness, or just hate their guts there is zero incentive to call out someone's work. Note that very little of what is published is obvious nonsense. But a lot has issues like "these energy measurements are ten times lower than what I can get, how on earth did they get that?" Maybe they couldn't or maybe you misunderstood and need to be more careful when replicating? Are you going to spend months verifying that some measurements in a five-year-old paper are implausible or do you have better things to do?

[DrScientist]

Sure - such direct contradiction is rare - call out was the wrong phrase - that mostly only happens which people try and replicate extraordinary claims.

Much more common is another paper is published which has a different conclusion in the particular area of science which may or may not reference the original paper - ie the wrong stuff get's buried over time by the weight of other's findings.

You could say that part of the problem is correction is often incremental.

In the end the manipulation by Masliah et al came out - science tends to be incremental, rather than all big break-throughs and I'd say any system will struggle to deal with bad faith actors.

In terms of bad faith actors - you have two approaches - look at better ways to detect, and looking at the properties of the system that perhaps creates perverse incentives - but I always think it's a bad idea to focus too much on the bad actors - you risk creating more work for those who operate in good faith.

[jakewins]

How is that correction mechanism supposed to work though? Do you mean the peer review process?

Friends in big labs tell me they often find issues with competitor lab papers, not necessarily nefarious but like “ah no they missed thing x here so their conclusion is incorrect”.. but the effect of that is just they discard the paper in question.

In other words: the labs I’m aware of filter papers themselves on the “inbound” path in journal clubs, creating a vetted stream of papers they trust or find interesting for themselves.. but that doesn’t provide any immediate signal to anyone else about the quality of the papers

[DrScientist]

> How is that correction mechanism supposed to work though? Do you mean the peer review process?

No. I meant somebody else publishes the opposite.

One of the things you learn if you are a world expert in a tiny area ( PhD student ) is that half the papers published in your area are wrong/misleading in someway ( not necessarily knowingly - just they might not know some niche problem with the experimental technique they used ).

I agree peer review is far from perfect, and there is problem in that a paper being wrong is still a paper in your publication stats, but in the end you'd hope the truth will out.

People got all excited about cold fusion - then cold reality set in - I don't think the initial excitement about it was a bad thing - sometimes it takes other people to help you understand how you've fooled yourself.

[AshamedCaptain]

I expressed the same idea here not too long - the value of any one individual paper is exactly 0.0 - and was downvoted by it, but I believe this is almost the second thing that you learn after you publish, and what seems to confuse the "masses" the most.

You (as a mortal, human being) are not going to be able to extract any knowledge whatsoever from an academic article. They are _only_ of value for (a) the authors, (b) people/entities who have the means to reproduce/validate/disprove the results.

The system fails when people who can't really verify use the results presented. Which happens frequently... (e.g. the news)

[CJefferson]

I'm in academia, and I think it has many good points.

The number one issue in my mind is competitors labs don't call you out. It's extremely unusual for people to say, publicly, "that research was bad". Only in the event of the most extreme misconduct to people get called out, rather than just shody work.

[po]

Yeah I don't think CRM is the correct thing in this case... I just think that there needs to be some new set of incentives put in place such that the culture reinforces the outcomes you want.