Very Interesting, though I still wonder if intelligence agencies can out perform betting markets (at least on questions with enough interest to be able to generate a liquid price).
That assumes that intelligence agencies are not utilizing betting markets.
In fact there was a big push in 2010 to build a betting market type system for analysis across the Intelligence Community. It is not used 100% of the time but has some measure of success where applied.
When I was in the IC I actually built a hypothesis estimation tool based on the Delphi method and a Bayesian updater to give probabilities for future events, crowdsourcing the votes from the community. Worked ok in our limited run, but needed a lot more work to implement really well.
There was an article in NPR a while back about the Good Judgment Project (which was run by Dr. Tetlock - mentioned in the article). In some cases, betting/prediction markets (& other similar tools) were beating intelligence analysts - http://www.npr.org/sections/parallels/2014/04/02/297839429/-...
I participated in GJP for a couple seasons. Honestly, I didn't do that well.
But I was approaching it very differently than I would if it was my job. Since the system that was set up gave high rewards to unpopular predictions, I just gambled on the few most unpopular that had at least some shot at reversing. It wasn't the smartest approach, but it was the most fun. If I was doing it for real, obviously I'd go a different way.
The people who did best, at least from what I saw, tended to ride waves of popularity on the more active questions, buying low and selling high.
What I did was like betting on a few biotech startups, what the best scorers did was like riding waves of the market leading stocks.
In the end, I'm not that sure it had much to do with actual prediction of events. Then again, neither did my approach. I guess I'm not sold on the version of prediction markets they were using.
Kinda depends on which platform you were on (GJP used several). Some were prediction markets while some were opinion pools, both of which are scored/rewarded differently.
Our company (Cultivate Labs) recently acquired Inkling Markets (a very early YC company that built prediction market software) and have been building a new version of the PM platform, which will hopefully address some of the risk/reward quirks.
The big difference they have is that they are playing a more strategic role than the tactical short term play in most market activities - in that their predictions will drive government action which will be a major factor in any outcomes. Makes more sense to test against successful opportunism than prediction
Could there be an element of self-fulfilling prophecy explaining why intelligence analysts, on whose advice governments decide their actions, have a better success rate at prediction than others who do not so closely influence government decision making?
Apparently, given that governments are cracking down on prediction markets. The conspiracy-theorist side of me could interpret it as intelligence agencies beating them quite directly.
In fact there was a big push in 2010 to build a betting market type system for analysis across the Intelligence Community. It is not used 100% of the time but has some measure of success where applied.
When I was in the IC I actually built a hypothesis estimation tool based on the Delphi method and a Bayesian updater to give probabilities for future events, crowdsourcing the votes from the community. Worked ok in our limited run, but needed a lot more work to implement really well.