[gingerlime]

> but they never really published much justifying it

Not that I'm trying to defend Optimizely (I'm not a huge fan, but for other reasons...).

I can't vouch for the quality either, but they did publish something about it[0] - that at least looks quite scientific. Happy to read any critique of course.

[0] http://pages.optimizely.com/rs/optimizely/images/stats_engin...

[norkakn]

Latex is a wonderful way to make a marketing paper look like a scientific one. It doesn't accurately describe the method, but that isn't really its purpose. It's a more technical description of the blog post, meant for people using the product to understand some of the tradeoffs and get more accurate results.

They are still having people make very fundamentally flawed assumptions about the data, which results in incorrect conclusions, and they are still not presenting the results in a way that people correctly interpret them. That being said, those are really hard to solve, and models that would try to correct for them would likely require a lot more data and be overly conservative for more people.

What are your reasons for disliking Optimizely?

[leo_pekelis]

Hi, this is Leo, Optimizely's statistician. If you're looking for a more scientific paper, maybe take a look at this one we wrote recently: http://arxiv.org/abs/1512.04922

Should have everything you would ever want to know about the method.

I agree with you that the problem of inference and interpretation between A/B data, algorithms, and the people who make decisions from them is a hard one and worth working on.

That said, I do think the two sources of error our stats engine addresses - repeatedly checking results, and cherry picking from many metrics and variations - did make progress in having folks correctly interpret A/B Tests. This did result in more conservative results, but the benefit was that the variations that do become significant are more trustworthy. I think this was absolutely the right tradeoff to make for our customers, and trustworthyness is a pretty important aspiration for stats/ML/data science in general.

Of course I did write the thing, so I'm not very impartial.

[gingerlime]

I was tempted to make a snarky comment about using LaTeX, but I'm not sure it's entirely fair. It doesn't seem like just a bunch of MarketingSpeak wrapped in LaTeX to be honest.

My issue with Optimizely are mainly how they essentially ditched us as (paying) customers. We were admittedly small-fish, but we were paying and were willing to pay more, but they switched to Enterprise-vs-Free without any middle grounds. Enterprise was way too expensive for us. Free didn't include essential features, so we were just stuck.

I ended up writing an open-source javascript A/B test client[0] (and recently also an AWS-lambda backend[1]), but it still has a way to go...

[0] https://github.com/Alephbet/alephbet

[1] https://github.com/Alephbet/gimel

[yummyfajitas]

It may be marketing but I was able to implement a sequential A/B test based on it. Admittedly, I did need to do some work beyond merely copy/pasting an algorithm, but all I really needed to do was read their paper and some citations. I do believe that this document does describe a viable frequentist test and my implementation of it worked pretty well.

Disclaimer: I do stats work at VWO, an Optimizely competitor.

(Also if you want to read our tech paper, here it is: https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technic... This describes our Bayesian approach, which we believe to be less likely to be wrongly interpreted by non-statisticians.)