[kqr]

> Depends on whether you're trying to move topline - or trying to move the "typical".

If this is an important difference, you should define the "typical" population prior to running the experiment.

If you take "typical" to mean "the users who didn't accidentally produce annoying data in this experiment" you will learn things that don't generalise because they only apply to an ill-defined fictional subsegment of your population that is impossible to recreate.

If you don't know up front how to recognise a "typical" user in the sense that matters to you, then that is the first experiment to run!