[Gianteye]

You've got it. The other half of the problem is that there's a chance that when you start flipping coins the first 100 flips will all turn up heads. Now, does this mean that the universe is bent and has begun preferentially treating Mr. Lincoln's head different than his backside? Does this mean your testing apparatus is biased? Is this an inherent property of coins? If you stop flipping at 100 it'd very tempting to conclude that this is the case.

The only way to find out is to do enough flips to eliminate the chances of your final result being influenced by statistical flukes. Measuring small differences, like trying to answer "does a coin preferentially land on one side vs the other?" usually takes hundreds of thousands of tests to guarantee you're seeing objective data, rather than seeing patterns in noise.

[hammock]

actually you ought to have selected n number of tests beforehand, rather than see the fluke and, after the fact, "continue testing" until it goes away.

the very moment you peek, your data is tainted from future testing.