[somenameforme]

You're also leaving out the biggest issue. Journals generally don't want to produce negative results. If you spend researching [shocking possibility] and it turns out that [shocking possibility] isn't true, you're not getting published. It motivates everything from HARKing [1] to outright data manipulation. By contrast if negative results were seen as valuable, then none of this is an issue.

On the other hand, it really is the case that there's just not much of any value in learning that [shocking possibility] is, as everybody would naturally expect, indeed not the case. And filling up limited journal space with such discoveries would seem to be counter-productive, at best. And when you have limited space/funding for researchers, one guy who keeps proving everything everybody knows to be false, to be false, is always going to be perceived as less valuable than one making [shocking discovery] [... which ends up being proven false years later].

[1] - https://en.wikipedia.org/wiki/HARKing

[naasking]

> If you spend researching [shocking possibility] and it turns out that [shocking possibility] isn't true, you're not getting published

But this simply isn't true in physics where negative results are very common. This is at least an existence proof that this can work, people just have to get their heads straight on what research means.

[staunton]

By "journal space" you of course mean journal prestige that isn't unlimited. The point of science journals is gatekeeping.

[bluGill]

The biggest problem is honestly obtained incorrect results. If you run 1000 experiments across 1000 labs. Few will statistically not notice a mistake and get a wrong result. That wrong result is then published as it is surprising.

[somenameforme]

I think there are some strong arguments against this. The first is numerical. Fields like social psychology are seeing replication rates as low as the twenties. And not just from low hanging fruit from but from journals like "The Journal of Personality and Social Psychology", which has one of the highest impact factors across all psychology journals, and a 23% replication success rate! [1] This [2] is a Google search for site:nytimes.com "Journal of Personality and Social Psychology". It's interesting seeing how many [shocking discovery]s, many which end up being shared on this site, come from this particular journal.

Furthermore, I think you can often see poorly done science in the papers themselves. They will use suggestive wording in surveys, unreliable sources for sampling such as Amazon Mechanical Turk, and maybe one of the biggest tells is measuring a large number of unnecessary variables. That does very little to further your experiment, but absolutely ensures you can p-hack your way to a statistically significant result. Another is ignoring such patently obvious viable confounding issues, that one can't reasonably appeal to Hanlon's razor.

[1] - https://en.wikipedia.org/wiki/Replication_crisis#In_psycholo...

[2] - https://www.google.com/search?q=site%3Anytimes.com%20%22Jour...

[QuesnayJr]

It can also be hard to judge whether replication failed because the result is bogus or replication failed because the replication team is themselves incompetent.

[edgyquant]

Did they follow the exact same steps claimed to result in something? Did it result in that thing?

The repetition team being incompetent sounds like a cop out. The researcher did a bad job and it’s on them to explain better etc in that case. No excuses, if it can’t be reproduced it isn’t taken seriously no exceptions

[QuesnayJr]

You have two groups of people. Either is equally likely to be incompetent.

[bluGill]

How do you know which? Both will point fingers at the other.

[edgyquant]

So just make sure someone unaffiliated has to be able to reproduce whatever research has been conducted. Tough luck if it doesn’t make it, that’s why you do your best to ensure you’ve verified it’s a real result.