[lupire]

It is absolutely unconscionable that "peer review" process doesn't include "review". Replicating results should be a hard requirement for publishing a conclusion. Anything short is merely a hypothesis with some suggestive evidence, a conjecture. Imagine if mathematicians cited published conjectures but called the proven results.

[sseagull]

This is a common refrain, but if this was a requirement science would absolutely grind to a halt. There is just too much science being done. And in some fields, what scientists publish is generally sound, if incremental (and maybe of dubious importance).

I think the tradeoff we have taken is "lots of science, mostly good" vs "little science, 100% correct". Similar tradeoff is taken with software & wikipedia, for example.

Or do you apply the same rigorous standards to, say, software? Should every piece of software that is publicly distributed be thoroughly reviewed for correctness?

[AndrewKemendo]

It probably should grind to a halt because the incentives for “research” are skewed to commerce to such a degree that the structure of the investigatory mechanism is foundationally flawed

[sseagull]

Easy to say, but good luck with that.

It's flawed for sure, but there's so much around you that comes from this system if you care to look for it.

[AndrewKemendo]

Honestly I don't find that process hard because I will not allow myself to live in a state of cognitive dissonance - conflicts in logic MUST be resolved. If there is no epistemic resolution possible, then we have nothing to say about this question with any degree of certainty. We can literally ignore all other information that claims to resolve it because we are not smart enough to actually define the problem.

For example we can make no certain conclusions on the "Hard question of consciousness." As a result there is nothing concrete about the concept of consciousness because it's not even a well defined enough question. Anyone attempting to make a claim are making it in an epistemic vacuum. So it doesn't matter who is talking about it, the state of knowledge of the concept of "consciousness" is so lacking that it doesn't even make sense to discuss.

In my view most of the persistent problems in humanity are in this class: we haven't defined measurements for the issue in question well enough to actually be able to approach a solution.

I always keep in mind that: "I'm probably wrong about my position"

If you are seeking data to confirm your idea then you'll be wrong more often than not

If you are seeking data to falsify your idea then you'll be less wrong more often

[Voultapher]

Hard disagree, set aside 30% of the budget from the get go by law and use that to let others you have no contact with try to replicate your wirk. I'm willing to bet, that if this were to be tried earnestly 99% of research would be compatible.

[sseagull]

You are imagining a solution to a very hard social problem. It is trivially simple on paper, but realistically extremely difficult to implement when you think it through.

The fundamental issue: what is in it for the reviewers in this case? Scientists are people, and you can't really legislate interest in replicating other people's work that is unrelated to your own, unless you really spend money on it.

And there's some bad news on that front: https://www.science.org/content/article/final-u-s-spending-b...

[Voultapher]

I'm there with you, it's a culture and social issue. That said the argument it's not fun seems quite weak to me. Scientists have to do source citation, formatting, peer review and many other things that aren't intrinsically done for selfish hedonistic reasons.