[rayiner]

But elections aren’t single events, they are 50 different events. 5% of the time the results in Michigan will be outside the error bars. But if the results are outside the error bars in Michigan, Iowa, Wisconsin, Minnesota, etc., and the polls all got it wrong in the same direction for each (overestimated Biden’s support) that’s actually statistically quite unlikely.

[saeranv]

Both 538 and the Economist forecasting models model errors as correlated - each state's vote share is not considered an independent event.

[mattkrause]

Wisconsin, Michigan, and Minnesota aren’t statistically independent though, so if a poll is wrong in one of them, it’s also likely to be wrong in the others too.

Correlated errors are tough!

[rayiner]

Maybe it’s semantics, but to me that means something is wrong with the model. It’s not just a matter of an unlikely result being happening some percentage of the time. There is some hidden factor that could have been adjusted for ahead of time.

[mattkrause]

Both, I think.

The model certainly ought to handle correlations, but the evaluation should take them into account too.

You can't treat the election as fifty independent replicates: a miss in both Minnesota and Wisconsin is clearly worse than one error, but it's also not as bad as (say) getting Wisconsin and Rhode Island wrong, where it's more likely that two separate errors occurred.

[rayiner]

I understand the point mathematically, I think I'm talking about more of how the presentation of the error term is unintuitive to me. It makes sense to me to say "the predications were right, Trump just got lucky" when he outperforms the 95th percentile error bars in a state. He just got lucky. But if he outperforms it in a bunch of states because the polls didn't account for the fact that Trump supporters don't answer pollsters, then it doesn't make sense to me to say "the predictions were right because they factored in the chance that the polls hard correlated errors." I get that you've quantified the possibility that the polls are wrong in a systematic way, but I don't think that's the kind of possibility people are thinking of when they hear "there is a 5% chance Trump could still win."