[masonhipp]

"The beauty of the market is that we allow people to be Bayesian" [...] "People come in with some prior belief, but they can also follow prices to see what other people believe and may update their beliefs accordingly [...] participants in the market could focus their bets on the studies they felt most sure of, and as a result, rough guesses didn’t skew the averages as much."

It certainly isn't a fool-proof method of increasing accuracy, and it does favor popularity of a theory over other factors, but overall it's probably a nice layer of data to consider adding to the mix.

[cba9]

> "The beauty of the market is that we allow people to be Bayesian"

Yes, this is the critical piece. The results of the Reproducibility Project were not remotely a surprise to Bayesian observers. People like Gelman have been pointing out for ages (and I mean back to the 1960s) that the prior probabilities in these fields is low and necessarily a lot of the results were false positives. With the rise of meta-analyses, it is possible to have informative priors for particular fields of psychology or for psychology as a whole, which would let you make much better predictions about whether a result was real. But you can't use these in papers - authors are heavily biased towards using procedures or flat priors which are uninterpretable or grossly overestimate the evidence, and if you try to use any of the informative priors or more advanced models, they'll nag you to death with a thousand objections and complain about double standards and subjectivity and how this time is different and (ironically) bias. So for the most part, there's not much to gain in academic research.

But in a prediction market, you don't have to listen to the self-serving excuses or explain your reasoning, and there's something to make it worth your while.

[notahacker]

Frankly, considering all the theoretical advantages to a prediction market in terms of encouraging people to influence the outcome in proportion to their confidence in the result, the fact that they were wrong 29% of the time, only 13% less than a simple blind binary survey of a pool of psychologists, isn't hugely impressive[1]. Scientists' goal, after all is to reach some generally accepted view of why a result turned out a particular way and in what circumstances a different result might be yielded, which is more nuanced information than can be conveyed by a simple market price. Asking a pool of scientists how confident they are about a particular survey's replicability and why - which is what the hypotheses are fundamentally all about - conveys more information than the prediction market. It's more useful to know that experts' doubts over replicability are linked to a specific survey design feature than it is to know the market's prediction of the probability of replicating it is 67.2% And financialisation of results might be actively unhelpful when it comes to debates over methodogical shortcomings of replication attempts and the pertinence of factors that were different between the studies. That's especially the case for a discipline like psychology where setting up absolutely identical conditions for a retest is a practical impossibility.

[1]and of course it hasn't been reproduced yet ;-)

[smt88]

It isn't fool-proof, but there is a lot of research into the phenomenon that groups of humans are pretty good at predicting outcomes (much better than most individuals). I forget the math behind it, but it makes a lot of sense mathematically.

Here's a book all about it: http://www.amazon.com/The-Wisdom-Crowds-James-Surowiecki/dp/...

[masonhipp]

Very true. There's another one floating around somewhere about how good we are at estimating the IQ of other people. Pretty interesting.