[jbay808]

> the true probability

Interesting expression.

After testing, it turned out that you flipped it and it landed on heads.

Does that mean that you've discovered that the "true probability" for that flip should have been 100% heads?

[_v7gu]

Of course not. If you're a frequentist you can say your best estimate is 100% heads with an unknown variance, and if you're a Bayesian you work out p(a|b) = p(b)p(b|a)/p(a) and update your priors (which will not give 100% heads). The more coins you flip, the better you can estimate the true probability

[SAI_Peregrinus]

IMO (I should have defined this) the true probability would require an infinite sequence of tests to determine.

[jbay808]

Ok, so let's imagine you build a servo-driven flipping machine to carry out an infinite series of tests and notice, after a thousand of them, that 99% of the time, the coin flip matches the orientation of the coin when it's loaded into the machine.

What have you learn about the coin's true flip probability?

[SAI_Peregrinus]

You've learned that the system of coin + machine has resulted in the same orientation 99% of the time. You can put some error bars on that, investigate the differences (did that 1% where it changed happen disproportionately with a certain side of the coin up?) and from that provide an estimate for whether the coin is fair. If the confidence intervals aren't small enough for you, you can do more experiments. The confidence interval will never be 0, until you've either done an infinite sequence of trials. (Only axiomatic logic can have confidence interval 0, and it doesn't make statements about the real world, only about the axiomatic system in use.)

[jbay808]

So, let's say that we continued the servo tests 1e99 times, with the coin loaded in each orientation equally. We measured 50.00% flips for heads and tails, and continue to see the 0.99 correlation with the initial orientation. The 1% of the time that the correlation doesn't match, it doesn't seem to show any bias for one side or the other.

So after an "infinite" number of tests, we continue to get 50.00% frequency of heads, but with an 0.99 correlation with the orientation when loaded into the machine.

Now I load a coin into the machine and ask you to name the true probability that the result is heads. I don't tell you the initial orientation, but I know it privately.

What's the true probability of heads? Our testing found precisely 50.00% frequency of heads. But are you still sure the probability is an intrinsic property of the system, rather than a property of your state of knowledge of the system?

We can continue the pattern; maybe the 1% error itself correlates to 0.99 with someone running the microwave in the kitchen. This drops the line voltage and causes the servo to impart a little less momentum to the coin, causing it to flip one fewer times on average. Neither of us have currently checked that the microwave is running... And so on...

[prmph]

But what do the error bars themselves mean? Are they not probabilistic in nature themselves?

Say you conduct a thousands trials and calculate the error bar based on the results. If you conduct a hundred such experiments (each consisting of a thousand trials) and one of the experiments violates the error bar, does that invalidate it?

[imtringued]

>What have you learn about the coin's true flip probability?

Nothing because you only tested the coin flip machine in aggregate. If you have a different throwing mechanism the results could be completely different.

[jbay808]

When I flip it with my hand, aren't I testing the coin-hand system in aggregate?

If nothing is flipping the coin, then what coin flips are we making predictions about?

[prmph]

But what if the apparent probability after n trials does not converge as n grows arbitrarily large?

If we observed such a system in nature, what would its "probability" mean?