[PaulHoule]

I didn't see a real Bayesian point of view in that article.

A Bayesian does not give you a probability estimate they give you a probability distribution for the probability!

Like in Star Trek Spock is always saying something like "Captain, we have a 15.31% chance of surviving this mission" which is a ridiculous example of precision without accuracy. [1]

If you observe a coin flipped 100 times and it came up heads 65 times it is not a crazy point estimate to say it has a 65% chance of coming up heads but this is just one sample and if you did it another time maybe it comes up 61 or 68 times. You are better saying that the probability distribution of the probability is β(65,35) or maybe β(65.5,35.5) or β(66,36) since that has the "error bars" built in, can be updated if you get more samples, etc.

[1] ... and you know he underestimates survival probabilities the same way Scotty overestimates how long it will take to fix the engines

[wosk]

The \pi_i in the paper is not the estimate of a latent parameter. It is the predictive probability of the event, which is a single number by necessity in a binary challenge. It's the integration of a distribution function which can contains very complex distributions: in my example something_you_believe can be a probability distribution.

So everything in the paper is distribution and when you forecast for a binary event, you give a number which is the expectation of that distribution. This is a probabilistic forecast.

If you were to give a probabilistic forecast for a continuous quantity, then yes you would give in a distribution, as in section 4.2

[edbaskerville]

Emphasizing this response. Bayesian models can always produce simple probabilities if you ask them to. E.g., given this data, what is the probability that the next flip is heads?

The fact that the model is represented as a distribution over Bernoulli parameter p doesn't contradict this: you just integrate over the posterior.

[PaulHoule]

I got dragged kicking and screaming into the Bayesian world in the early 2010s when I was trying to bring back the old AI and came to the conclusion that the real problem with the old AI was lack of a systematic approach to reasoning about uncertainty and not the cost of knowledge base maintenance (e.g. that could be brought down by orders of magnitude, the value created by knowledge bases could be brought up)

One problem is aggregating information over multiple steps in the reasoning chain the other problem is the powerset problem. The point probability estimate from integrating over the posterior wasn't useful for the first, so I didn't want it. The second problem is impossible in theory but possible in practice, as the existence of intelligence proves.

[Joker_vD]

We've got a coin here! Let's ask a frequentist and a bayesian what they think about the probability that flipping it would make it land heads up?

Frequentist: How would I know? I haven't seen it flipped once, nor do I know how you've selected it from all the other coins that exist.

Bayesian: It's 50%!

Then we flip the coin 10,000 times and observes that it landed up heads exactly 5,000 times.

Frequentist: Huh, that's weird. You see, if the probability was 1/2, the expected deviation from the mean in this case would've been 50, so I'd expected to see either about 4'950, or 5'050 heads... still, MLE provides the answer of 1/2 bu-u-ut...

Bayesian: It's 50%!

This two-strawmen thought experiment clearly demonstrates the superiority of the Bayesian approach in learning useful information from the real-world observations.

It's really kinda shame that both personal certainties and physical probabilities follow the same algebraical rules while having entirely different nature; most of the time, you are not very interested in how much is someone is certain of some outcome, you're much more interested in the actual outcome or at least the actual probability of that outcome. Granted, most of the time you can only readily access someone's certainty of an outcome, but this is just a proxy for the quantity you're actually interested in knowing.

> You are better saying that the probability distribution of the probability is β(65,35) or maybe β(65.5,35.5) or β(66,36)

s/probability/your personal certainty/g. The probability of the coin landing heads up is what it is, and it usually doesn't depend on any of your knowledge.

[bryanrasmussen]

>and you know he underestimates survival probabilities the same way Scotty overestimates how long it will take to fix the engines

NOT SPOCK!!!

[clickety_clack]

Was Spock poorly calibrated?

[PaulHoule]

If he was well calibrated there is no way they would have made it through 79 episodes!

[bryanrasmussen]

Spock had not realized that James Kirk emitted a psionic reality distortion field through higher dimensional "luck", if he had he would have been a bit more relaxed.

[oliculipolicula]

To his defence. in the trekkie timeline, neither Vulcans nor humans had discovered the Metropolis algorithm, so Spock had (for sure) nothing with which to calibrate that