Hacker News new | ask | show | jobs
by edmundsauto 2109 days ago
How do you know that it wasn't caused simply by selecting for more serious trialers? What makes you think the cause was "committing", and not a selection thing?
3 comments

Because the conversion rate was higher from no sign-up to paid. So we were probably selecting more serious trailers, but they were then more likely to pay as well. If it was just the conversion rate from sign-up to paid then you would be right, but we looked at the conversion across the entire funnel.
"More serious trialers" will be more likely to commit by entering an email; people who are not willing to enter an email are obviously not very serious trialers. It's just two different ways of saying the same thing.
No, one implies a causality that the other doesn't.
"More serious trialer" is not a fixed attribute or category that you can select for. A person fits this description based solely on their behavior; they become a more serious trialer by the act of putting in their email address. From that point the causality is the same.
Yes - my point was that the narrative of "why" isn't useful. A/B tests tell you what happened. They do not provide insight into why. Yet everyone has a story they tell to fill in the why, and these stories are harmful to making good decisions.
It's not a causality, it's a pre-filter. So, of the people who passed filter A more are likely to pass filter B.

Today's XKCD was basically about this (in the Alt text)

https://m.xkcd.com/2357/

desktop version: https://xkcd.com/2357/
> We AB tested this