[chillacy]

This is the track record for Metaculus (the chart at the end) showing historically a 10% swing in predictions of 50% historically by aggregated forecasters on the site: https://www.metaculus.com/questions/track-record/

Granted this tells us nothing about the accuracy of any one particular question, but on aggregate Metaculus is pretty well calibrated.

That said, there is a longstanding academic debate between "forecasting can tell us useful things" (Telock) and "black swans will ruin your life" (Taleb).

[AussieWog93]

>Granted this tells us nothing about the accuracy of any one particular question

I think this is where my skepticism comes from.

Just because one can make certain predictions with reasonable accuracy (e.g. Scott Morrison will lose the upcoming Australian election, Joe Biden will die in office, Elon Musk will be the richest man in the world by 2030), it says nothing about their ability to predict other questions with far less information (such as if and when the CCP will invade Taiwan).

[chillacy]

That's a fair take. In research about forecasting, they do indeed find that timeline greatly impacts forecasting accuracy.

Even if we were to fix this particular bucketing issue with a large enough dataset of foreign policy predictions X months into the future... one could still say this tells us nothing about this specific question because... maybe even among the geopolitical questions, this question is somehow different in some way which makes it noncomparable.

Or by analogy it would be like we were flipping what we think are coins and someone snuck a dice roll on one of the tries. Maybe the Taiwan question is the dice roll and therefore not subject to comparison with other geopolitical questions X years out.

This is of course afaik an unknowable problem, and Taleb would agree that we should just assume these predictions are tea-leaves.

At the same time it may be a valuable source of info (certainly imo more empirical than pundits and the news)

For consideration:

https://english.elpais.com/science-tech/2022-03-26/is-it-pos...