Hacker News new | ask | show | jobs
by benkuhn 2044 days ago
This is a shoddy analysis.

1. There's a statistical gadget specifically for doing this—a "scoring rule" [1] which is a principled way to compare different probabilistic predictions. A bunch of scatterplots of random quantities against each other are... not that.

By comparing only binary win/loss predictions instead of probabilities, like in the first chart, you throw away almost all information contained in the probabilistic estimates—if Democrats win a state, there's no bonus for predicting (say) 95% Dem instead of 55% dem.

It's plausible that 538 would actually win under a proper scoring rule, because betting markets were underconfident (relative to 538) in deep dem/rep states (predicting e.g. <95% Dem win in VT, vs 538's >99%). [2]

2. The calibration analysis assumes that different state win/loss rates are independent, but that's really untrue: 538's predictions were specifically not independent because they assumed polling errors were correlated between states.

3. Many of the other scatterplots look outlier-driven and don't include r^2 or p-values. With so few datapoints, it's unclear if they are meaningful at all.

[1]: https://en.wikipedia.org/wiki/Scoring_rule

[2]: Maybe we should cut prediction markets some slack here because liquidity constraints make them inaccurate for small probabilities. If that's the article's position, though, they should address this instead of just... not using a scoring rule.

6 comments

You might be interested in this blog post, which _does_ use a scoring rule: http://zackmdavis.net/blog/2020/11/scoring-2020-us-president....
Oh, great! And the prediction market did not beat polls (it tied with them on swing states and beat them on safe states, as I speculated).

I guess between this and the other commenter who made money betting 538's model on predictit, I consider this pretty thoroughly debunked.

You can’t do the analysis on PredictIt, they limit market size heavily.
Do you know of any similar sites that don't limit bet size?

I saw some sports betting sites were offering odds, but not really a true market like predictit I think

Betfair seems to offer most of the same contracts as PredictIt as two-sided markets with lower fees and no cap on bet size or number of participants.
They are geolocked for some features, however
augur
I still kinda feel like the market beat the polls. Kinda.

In order to interpret the polls, you need top statisticians with lots of domain knowledge. In order to interpret the betting market, you simply read the odds and are done.

If you went to the market and you bet that the polls would be correct, you made money.

The particular market available to US residents has separate issues about "reading the odds" where events that strictly contain other events are often priced multiple pennies cheaper, the total number of expected presidents is often ~1.1, events with probability at least 0.9 are almost always underpriced, often by a nickel or more, etc.

I’m interested in the liquidity constraints theory - Betfair has ~$1bn matched for the most recent election. At present there is ~$700k bid and offered at the market for both candidates (the market is still open). Could you expand on your thinking?
Sorry, that was imprecise. My impression is that, at least on some prediction markets, transaction fees (and maybe also inflation?) make it low-return to buy high-probability contracts. I don't bet on prediction markets myself so I may be wrong about this though!
How would the fees make it any less profitable for high probability contracts vs low probability contracts? And could you expand on the inflation theory? Appreciate the insight!
If you look at it in terms of risk-adjusted returns (Sharpe ratio), a fixed transaction fee costs more Sharpe on a low-variance bet than it does on a high-variance bet. The variance of balanced contracts is higher than the variance of unbalanced ones (p(1-p) is concave with maximum at p=0.5), so after adjusting for risk the transaction fee is more significant for the unbalanced contract.
what’s a ‘balanced’ contract?
I would be guessing that it's one where equal amounts of money are bet on each side/ there's a 50% probability of the result going either way.
A market where expected returns are 1, I. e. the implied probabilities sum to 100%
High probability contracts require you to put up a lot of capital to make a small profit. This would be fine for a bet that has a quick turn around, but prediction bets often require the bet to be placed well in advance of the event, which means the capital is locked up for a long time.... it is basically a low interest loan to the bookie.
Yep cost of capital != inflation
Opportunity costs on long term high probability events eats up significant portions of the value of winning.

Add withdrawal fees and you could easily lose money while making an accurate prediction.

Sure but cost of capital != inflation. And I understand the impact of fees but I’m wondering how they could influence high likelihood bet profitability more than low likelihood bet profitability?
Let’s suppose your paying a 10% withdrawal fee on winnings and 0% fee on principle. If your payoff is 10% and you're also losing out on 2% to opportunity cost. Net winning is (10% * 0.9 - 2%) = 7% but your withdrawal fee is effectively reducing your winnings from 8% or 7% a loss of 12.5%.

On the other if the bet paid 100% of your money you would net (100% * 0.9 - 2%) = 88% and your fee dropped you from 98% to 88% a net loss of 10.2%.

PS: These numbers get much worse if the withdrawal fee includes the principle. Then you would be have a net loss of (110% * 0.9 - 2%) = 3% on the first bet and a net gain of 78% on the second.

yep, predictit takes a 5% fee on profit + 10% on withdrawals
Other way around. 10% profit 5% withdrawal.
They also only looked at a single election where Republicans outperformed polls across the board. If you look at the 2016 election, 538 was more bullish on Republicans than betting markets, and came out ahead.
In addition to just using Brier scores (or alternatives) to evaluate their performance, I'm also interested in seeing how 538's underlying Monte Carlo models shift based on x% of error/uncertainty? (for example instead of evaluating a polling advantage as a single value K within a single simulation rather consider it a normally distributed variable).
Also betting markets are highly influenced by both polls and published election models, so it’s hard to say that they ‘beat’ them while also being influenced by them.
Not to mention that 538 gave Biden a ~90% chance to win right before the election, whereas betting markets gave Biden a ~60-65% chance to win.

So if you go by the most important metric of what was trying to be predicted, 538 was far more accurate for the one giant data point.