Hacker News new | ask | show | jobs
by kqr 1844 days ago
I'm not (yet) using a scoring rule for my work-in-progress uncertainty test[1] of calibration, but only Beta posteriors, which are also a neat way of presenting the result of many predictions.

I am slightly more fond of log scoring than the Brier score, though, for the reason mentioned in another comment: being somewhat wrong is often worse than being very right, and should be penalised harder numerically.

[1]: https://static.loop54.com/uncertainty-test.html

(By the way, I build this to practise myself -- but I ran into a problem: I know the answers to all propositions, having written them myself... if anyone wants to contribute propositions, please contact me and I'll ask for them in a specific format so I can blindly paste them without knowing the true ones.)

1 comments

This looks quite cool - would you be able to drop me a line at datarecipes@pm.me?