[coolcase]

Sounds like a variable cost experiment. Each observation cost x$. Like an A/B split on Google ads. Why keep paying for A when you know B is better already.

[nialse]

Small samples have more variability than large samples and thus more often show spurious large effects.

[coolcase]

So you end up with a higher threshold for confidence at p<0.05 ot whatever you want p to be under. Comes out in the maths!

Toss a coin 10 times comes up heads 10 times. There is a 1 in 2^10 (approx 1000) that happens by chance for an unbiased coin.

I'm convinced it is biased.

20 times I am freaking convinced.

I don't need another 1000 tosses.

[azan_]

It’s more like you are supposed to toss 1000 times and after 500 tosses you get a lucky streak of 5 heads in a row and then decide to end experiment and conclude that coin is biased.

[coolcase]

Oh yeah. Don't do that! Look at all 500 tosses.

[rrr_oh_man]

Google Optimize used to tell you to let an experiment run for one-two weeks (?), exactly because early strong results tend to not don't hold up in the long run.

-> https://en.wikipedia.org/wiki/Regression_toward_the_mean

[dr_dshiv]

Seasonality effects, too