Ask HN: What is the best way to track user fidelity?

[marcelocamanho]

Hi.

I develop a price comparison website, and we track user conversions (pay by click model).

What is the best way you guys suggest to track user fidelity? That is, if a given feature is making users use the website more? This could be interesting if used together with A/B testing.

Any ideas? Don't want to use any invasive user tracking stuff though.

[patio11]

Cohort testing. For users signing up between T+0 and T+2 weeks (or whatever), you randomly assign them to N test groups. Each of the N groups has some combinations of features turned on or off, controlled via feature flags. You collect aggregate statistics over time of the conversions in each test group. Presumably, if you want to track outgoing clicks, you have the page intercept clicks and fire a Javascript event with an anonymous ID number which your backend will score as a click for the appropriate test group. The technical implementation of this will take you between 2 and 5 days or less.

The math gets hairy and you need a metric truckload of time/traffic to do anything useful with this, especially as you get to many types of conversions of interest or many different variables under consideration, as the nature of statistics means you'll always be able to find some correlation if you look in enough places. "Look doing Signup Form B and disabling private messaging decreases friend requests by 18% in month 3 with 99.999% probability!" [+]

+ This is not actually what the math is telling you.

[marcelocamanho]

I will look into it, thanks!