[carlmr]

I think what kevinwang is getting at, is that if you A/B test with a static version A and enough versions of B, at some point you will get statistically significant results if you repeat it often enough.

Having a control doesn't mean you can't fall victim to this.

[ricardobeat]

You control statistical power and the error rate, and choose to accept a % of false results.