[wosk]

> What does this even mean...?

You are making the same point that I did, if you read the rest. 0.7 at 80% is an intermediate step in formulating a forecast, and the point is to show that you cannot stop there and have to include everything in your number, like you have said. In your words, the intermediate steps are what you list (trust in weather forecasts, past experience,…).

[mitthrowaway2]

No, what I'm saying is that 0.7 is the ending point of the calculation, and there is no 80% associated with it. The 0.7 number already factors in my uncertainty in all of my knowledge about whether it will rain tomorrow. Saying that you're 80% sure of your 0.7 probability forecast throws a TypeErrorException.

[wosk]

yes and in my example the ending point of the calculation is 0.58.

I make an intermediate step explicit in this derivation because it is important for the understanding of the problem.

This kind of intermediate step happens a lot in forecasting competitions, where participants are asked for their forecasts and their confidence in their forecasts. I want to show here that you need to include that in your forecast (all you belief) and not keep it separated.

[mitthrowaway2]

But again this seems like a TypeError.

If you're asking me for my forecast and my confidence in my forecast, I would say this:

Forecast: It will rain tomorrow. Confidence: 0.7

If you say "No no, I mean what is your confidence in the 0.7 number" then I have no idea what you're talking about. 0.7 is my confidence. It's not valid to attach a confidence to that number. When the next day comes, we simply find out whether it rains or does not rain, and whoever put the highest probability on the correct outcome wins.

If you're doing this with intermediate steps then I have to ask: where did the 0.7 probability come from as an intermediate step, and what is the epistemic meaning of the 0.8 attached to it? If you only felt 80% sure that you were 70% sure that it would rain tomorrow, then what is the proposition that you are 70% confident about?

[mitthrowaway2]

Addendum: The closest thing I can think of that matches the GP's description is conditional probabilities.

For example, if it rains tonight then I am 70% sure it will rain tomorrow too. I am also 80% sure it will rain tonight. And I am 100% sure that if it doesn't rain tonight, it also will not rain tomorrow. Then I can chain these together: The chance of rain tomorrow is P(Rain tomorrow) = P(Rain tomorrow | rain tonight) * P(Rain tonight) = 0.8*0.7 = 0.56.

You can assign conditional probabilites to the correctness of your model itself. Then the phrasing is more like "I am 70% confident that it will rain tomorrow, conditional on my understanding of meteorology being correct" and "I am 80% sure that I understand meteorology correctly". Then you also need to add in a term for P(Rain | not understanding correctly).

[wosk]

yes exactly, it's a conditional forecast. I intended: P(Rain tomorrow | my model is right) = 0.7 p(my model is right) = .8 p(my model is wrong) = .2 p(rain tomorrow | my model is wrong) = something_you_believe_about (= 0.1 in my example) 0.80.7 + 0.2 0.1 = 0.58

you may replace "my model is wrong" by e.g " my certainty that no one is using cloud-activation technology".

Now, I'm not native in English and I see how someone (especially with a math background) might find this sentence confusing:

> you are predicting something (e.g rain tomorrow), then if you believe it will rain with probability 0.7 but you are 80% sure of your belief, you won’t say 0.7; you will say something else: 0.8 × 0.7 + 0.2 × something_you_believe = 0.58. Coherence forces you to collapse your uncertainty into your probability at each forecast.

Turns out it's very hard about math non-formally for generalist audience and might not have done the best job about it. It should read

> you are predicting something (e.g rain tomorrow), then your model believe it will rain with probability 0.7 but you are 80% sure of your model, then you won’t report 0.7; you will report : 0.8 × 0.7 + 0.2 × something_you_believe = 0.58. Coherence forces you to collapse your uncertainty into your probability at each forecast.