[thaumasiotes]

> for instance if a prediction market says there’s a 60% chance of an event occurring, and it doesn’t occur, was the market right or wrong? Well it’s hard to say - certainly the market said the event was more likely to occur than not, but only just, and who knows? Maybe the event _only just_ occurred, and very nearly didn’t!

For most events like this, you'd want to see the market spike to 0% or 100% as the deadline approached. And in particular for an event that happens, you want to see the spike to 100% before it happens. Remaining at 60% until after the fact is wrong because the occurrence of the event becomes more certain as it gets closer.

Being "well-calibrated" as you describe is a very bad quality metric in the sense that two sets of predictions can achieve the same calibration profile while differing markedly in quality. The farther the predictions are from 50%, the better they are, but your calibration metric doesn't take this into account.