[awesomebob]

A question for Patrick, or for anyone else with experience implementing A/B testing from the early stages of a service/product:

When and where do you start? If you start too early, you don't have enough traffic to be statistically significant.

In the article, Patrick said that he started with the trial signup form in his 3rd year.

Most of the advice on A/B testing that I've found is understandably aimed at people with existing business/products that can massively benefit from it. Does anyone have any more material about how to get started from the early stages?

[gk1]

I've shared some thoughts on this previously: http://www.gkogan.co/blog/test-big-changes/

TL;DR - Test big, drastic changes instead of fiddling with button colors and headlines. Examples of drastic changes:

- Entirely different homepage with different messaging.

- Change "Features" page to "Benefits" page and change its content accordingly.

- If you're offering a free download or free product, test asking for an email first.

- If your SaaS has a long signup form, test allowing people to jump right in with just an email address.

- If you're sending a robotic "welcome" email, test sending a very short personalized email instead.

[ssharp]

This has been my mantra as well. Test big and you'll likely fail or win a lot faster.

I'd also suggest making sure you approach testing by starting with a hypothesis and then testing that hypothesis. By voicing or writing down your hypothesis, you will be more likely to avoid the "stupid test trap", a phrase I use to describe running tests like changing button colors -- what hypothesis is changing button colors trying to prove? That people like red more, "for reasons"?

As mentioned above, each A/B test you run has a serious opportunity cost. I'm fortunate enough to have thousands of people going through my sales funnel daily but, even then, a test will still take several weeks or months to validate and anything approaching a change of 5% or less will really eat away at your ability to test more meaningful things.

[btilly]

See https://news.ycombinator.com/item?id=12541290 for one rule of thumb and methodology.

My experience is that most tests will come out under 5% differences. But most organizations will have some test that they could run with a greater than 15% win.

If your business scale does not allow you to run tests and detect the wins you hope for, you're better off following other people's best practices than you are trying to discover best practices from running your own tests.

Also note that most businesses have a "conversion funnel" where there are a number of steps from visitor to getting paid. If your business is big enough, you want to focus on getting paid. If your business is too small to get results that way, you should get started with just the first step in the funnel. That's what Patrick did with the trial signup form.