|
|
|
|
|
|
by chashmataklu
626 days ago
|
|
|
TBH depends a lot on the business you're experimenting with and who you're optimizing for. If you're Lime Bike, you don't want to skew results because of a Doordasher who's on a bike for the whole day because their car is broken. If you're a retailer or a gaming company, you probably care about your "whales" who'd get winsorized out. Depends on whether you're trying to move topline - or trying to move the "typical". |
|
|
If this is an important difference, you should define the "typical" population prior to running the experiment.
If you take "typical" to mean "the users who didn't accidentally produce annoying data in this experiment" you will learn things that don't generalise because they only apply to an ill-defined fictional subsegment of your population that is impossible to recreate.
If you don't know up front how to recognise a "typical" user in the sense that matters to you, then that is the first experiment to run!