[fredophile]

> Getting more people into the bucket of "people who installed a custom theme" doesn't actually help your bottom-line conversion rate until you can establish unequivocally that the people you've added to the bucket convert at the same rate as the bucket did before they were added to it.

This is incorrect. Let's say we have two buckets, A and B, where bucket A is people who haven't used a feature and bucket B is people who have. If some testing shows that people in bucket B convert far better than people in bucket A you make a change to get more people into bucket B. The change is successful if more people end up in bucket B and bucket B still outperforms bucket A and the total contribution coming from bucket B is higher. Bucket B doesn't need to perform equally well before and after the test.

Using the numbers from the link bucket B was performing at 10x the conversion of bucket A. Out of 10,000 users bucket B contributed 70 subscribers. After the changes bucket B received almost five times the users as previously and converted at four times the rate of bucket A. Out of 10,000 users bucket B contributed 884 subscribers. If the change had no impact we'd expect more users in bucket B but not more subscribers. These numbers aren't really accurate since I took them from the diagrams in the article and they change what they're looking at between the two. To be correct you need to compare the same thing before and after changing a feature.