[dragonwriter]

> His tone comes off as very defensive about his own work rather than constructive criticism of the Economist model.

Most of the linked thread isn't about his own work or the Economist model, but about the false description of their forecasts as being pretty close, made as part of the (true) description that fairly small changes in intermediate results in the Economist model would lead to the same bottom line forecast as Silver's model.

[iainmerrick]

I don’t agree with the “fairly small changes” part -- it looked like they were talking about 1% change in mean estimate and something like 25% change in variance. Those sound small, but they’re really not!

Or I guess another way to put it is, given the highly polarised two party system in the US, elections tend to be fairly balanced and small differences have an inflated impact on the results.

But whether you see the difference as small (1% difference in popular vote) or large (3x difference in chance of victory), the models are different, and one of them must be better, although it’s probably impossible to tell which it is!

Comparing models across multiple elections and calculating the Bayesian regret is one way to do it. The models get tweaked each election so this isn’t exact, but it could give a sense of the skill of each forecaster.

Does anyone have links for previous election forecasts from Morris and/or Gelman?

[dragonwriter]

> Comparing models across multiple elections and calculating the Bayesian regret is one way to do it. The models get tweaked each election so this isn’t exact,

As long as the same inputs aren’t fundamentally unavailable (and even then if the model has systematic handling of missing data, though the validity of that comparison is less clear) you can run the tweaked model on past elections (the main problem there is since those are probably the data used to generate them, it rewards overfitting; you might do better by running them against past elections with differing sets of random dropout of input data points [individual instances of particular polls, etc.] to mitigate that.)