[bryanh]

I rarely see many people take into account the opportunity cost of letting a really close A/B test reach 99.99% confidence when the benefit is by definition very marginal (that's why its taking so long, right?). I mean, is it really that bad to go on "close enough" results and move on to bigger and better tests?

[hammock]

In academia, we used p values of .95 or greater. I was taught that in business, though, the rule of thumb for making decisions is more like .8, and that's typically the standard I use as well.