[dbroockman]

Another one: don’t program your own AB testing framework! Every time I’ve seen engineers try to build this on their own, it fails an AA test (where both versions are the same so there should be no difference). Common reasons are overly complicated randomization schemes (keep it simple!) and differences in load times between test and control.

[giraffe_lady]

I don't keep that up with it but it seems like the ecosystem has kind of collapsed the last few years though? Like you have optimizely and its competitors that are fully focused on huge enterprise with "call us" pricing right out the gate. VWO has a clunky & aged tech stack that was already causing problems when I used it a couple years ago and seems unchanged since then.

If you're a medium-small business I see why you'd be tempted to roll your own. Trustworthy options under $15k/year are not apparent.

[HWR_14]

Shouldn't AA tests fail a certain percentage of the time? Typically, 5% of the time?