[rayiner]

Maybe it’s semantics, but to me that means something is wrong with the model. It’s not just a matter of an unlikely result being happening some percentage of the time. There is some hidden factor that could have been adjusted for ahead of time.

[mattkrause]

Both, I think.

The model certainly ought to handle correlations, but the evaluation should take them into account too.

You can't treat the election as fifty independent replicates: a miss in both Minnesota and Wisconsin is clearly worse than one error, but it's also not as bad as (say) getting Wisconsin and Rhode Island wrong, where it's more likely that two separate errors occurred.

[rayiner]

I understand the point mathematically, I think I'm talking about more of how the presentation of the error term is unintuitive to me. It makes sense to me to say "the predications were right, Trump just got lucky" when he outperforms the 95th percentile error bars in a state. He just got lucky. But if he outperforms it in a bunch of states because the polls didn't account for the fact that Trump supporters don't answer pollsters, then it doesn't make sense to me to say "the predictions were right because they factored in the chance that the polls hard correlated errors." I get that you've quantified the possibility that the polls are wrong in a systematic way, but I don't think that's the kind of possibility people are thinking of when they hear "there is a 5% chance Trump could still win."