[thehappypm]

Hey! Cool product! I have run many A/B tests in my career, and agree that it really does take ages to get statistically relevant results, and most results don't really pan out at scale. (Though in my experience, a new button color might drive conversion just by catching the user's eye as something new, but this doesn't stick around).

How do you get A/B test results faster? New math? Or just more BS? ;)

[tipiirai]

The secret is on Volument's ability to measure a bunch of "leading indicators", like the ratio of first-time visitors who stayed on the website more than 7 seconds (which indicates a better first impression). For every key conversion event at the bottom of the funnel (such as "signed up"), there are thousands of leading indicators to take insights from. By utilizing these top-of-the-funnel metrics you'll know whether the new version better in a matter of days — even hours on a very busy site.

Another fact is that Volument is a full-blown analytics software it already knows all the data for the baseline variant "A", so you only need to collect data for the "B" variant after a specific timestamp. Further cutting the measurement time by 50%.

More about the topic here:

https://volument.com/learn/

[rokob]

That sounds a lot like multiple comparisons type I error as a service.

[tipiirai]

Can you elaborate this, please? An example would be best.