[crazygringo]

I've tried to understand this (obviously quite angry/ranty) article and cannot actually figure out what data it has.

It seems to not be based on actual replication results, but predicted replication results? But then the first chart isn't even predictions from the market, but just the author's predictions?

The author clearly has a real hatred for practices in the social sciences. But I don't see any actual proof of the magnitude of the problem, the article is mostly just a ton of the author's opinions.

Is there any actual "meat" here than I'm missing? Or is all this just opinions based on further opinions?

[leftyted]

It's based on this: https://www.replicationmarkets.com, which is linked in the first paragraph of the article.

Per https://www.replicationmarkets.com/index.php/rules, volunteers are predicting whether 3000 social science papers are replicable. According to the rules, of those 3000 papers, ~5% will be resolved (i.e. attempts will be made to replicate). According to the article, 175 will be resolved. It's unclear to me who exactly will do that work but I would guess it's people behind replications markets dot com (they are funded by DARPA). The rules say that no one knows ahead of time which papers will be resolved so I assume the ~5% (or 175) will be chosen by random.

The data in the article seems to be based on what the forecasters predicted, not which papers actually replicated (that work hasn't been done yet...or at least hasn't been made public). The author of the article is assuming that the forecasters are accurate. To back up this assumption, he cites previous studies showing that markets are good at this kind of thing.

The tone is ranty but, by participating in the markets, the author is putting his money where his mouth is.

[brandmeyer]

I think you're right. Take a look at the before/after curves for "this is what the predictions look like after the papers".

The before curves are gaussian+ distributed and pessimistic, but the after curves are all distinctly bimodal (or worse). This suggests that some population of the participants were broadly pessimized by their surveys and another population was broadly optimized by their surveys.

This could instead be a measurement of how people's trust in science is predicated on how well it matches their own prior beliefs.

+ A sharper eye shows they aren't quite bimodal in the prior belief. Even in those cases, the separation between the modes gets much wider.

[jonnycomputer]

No, you are exactly right.