[throwaway2245]

I'd argue that 538's model likely overestimates uncertainty, because it is a benefit to them both ways.

Either they spin it that they were perfectly correct in those 95% of cases where they get it right, or they spin it that they were the least wrong (and therefore the most correct) in the other 5% of cases where everyone gets it wrong.

[chalst]

At no point did 538 make such a high forecast: their highest forecast was 88%.

There's a joke that you should always express 60% confidence in your predictions, since if the prediction pans out, you can claim to be right, but if it fails, you can bring attention to the "two times out of five wrong" part.

[throwaway2245]

Yes - thanks for explaining this better.

[ghaff]

538 has actually written about calibrating past forecasts. https://projects.fivethirtyeight.com/checking-our-work/

[throwaway2245]

Their calibration is good in the context of this article, but there is a reason this article doesn't use their Presidential election model - there isn't enough data to do this calibration here.

[patmorgan23]

Or maybe predicting the out comes of elections are really hard.

[panarky]

So you're saying that 538 wanted an uncertain outcome and hacked the model design and parameters to deliver the outcome they wanted?

[throwaway2245]

I'm not saying this was intentional, but there is a very high incentive for them to deliver this type of outcome.

There is not enough data to back-test their model on a single election which happens every 4 years, so the claim that a 60% prediction is fundamentally very different from a 95% prediction is statistically dubious.

[frgtpsswrdlame]

It's not a benefit to them both ways, it's only useful to them if there's a substantial amount of those 5%s in which case they're probably right that everyone else is underestimating uncertainty.