[Dyac]

So, I am exactly your target market. I run optimisation at a mid-size global company with a few websites (ranging from 100k to 100m sessions per year).

I do A/B testing day in day out. While it has its issues, I think completely dismissing it the way you do on the site makes me question everything else the site claims.

I looked at the live demo and it says about "significance", so I guess you're using a frequentist approach. The sample sizes are to small though, and I think you have SRMs, and it seems like the start and end times for the variants don't match. Also, in ecom it's bad practice to run any test or do much analysis on incomplete periods (eg a pay cycle).

Why aren't you using bayesian stats?

How can I compare test variants? We use monetate or SFCC a/b testing. Can it hook off events like GA can?

[fuzzymind]

The major difference between traditional A/B testing and Volument is that the comparisons are based on _cohort analysis_. It takes two groups of visitors and places them side-by-side starting from the very first visit: how they gradually build awareness and interest (metrics from the first visit) before taking action (on the first visit or on later visits). Then you take a cutoff day (say Day 7) and compare what has happened (how much retention, conversions, sales, virality) before that day. The whole "traction" so to speak.