[AftHurrahWinch]

This: https://vincentarelbundock.github.io/Rdatasets/doc/dslabs/po...

[jstanley]

That's circular reasoning.

My model says there's a 70% chance of Foo, and that is actually accurate. How do you know it's accurate? Because my model said so.

It might have been accurate! 30% probability events happen 3 times out of 10. We just have no way to know if it truly was accurate.

[softwaredoug]

You can “know” because we have decades of polling and election outcomes.

It’s not black and white “know”, 70% is the mean of a probability distribution.

It’s more accurate to say, this model predicts “Foo” because historically polls like this favor Foo 70% of the time. But these are probabilities and have wide errors. It’s on the reader to have a level of statistical knowledge.

These are more handicaps than “predictions”. The same way we predict whether it might rain tomorrow, who might wins tomorrows game, without a Time Machine.

[jstanley]

To say that polls are accurate in general is fine!

To say that a specific probability given by a poll was accurate is meaningless, there is no way to know.

[softwaredoug]

You can test your model on past elections

IE X% of the time my model predicted the right result in this particular election.

You can test weather forecasts on the weather in the past.

You can test your model on a sports game on past games.

We do this all the time in many fields. With different degrees of certainty (error bars are small or large).

The entire basis of machine learning and predictions people use in everyday life is based on this assumption

[jstanley]

That shows that your predictions are well calibrated in the aggregate. There is no way to know whether any one particular prediction was accurate.