[Eridrus]

I think the discourse is converging on A/B testing being a big company thing, where I think it is very useful.

Not because it finds new knowledge, but because it keeps your product teams honest.

It's really easy to delude yourself and others about your project when your promotion is on the line, and A/B tests let you actually evaluate whether the change helped or not.

At small companies, you're not trying to find 2% effect sizes, anything that small is already a failure, so you don't need statistics to tell you what worked.