|
|
|
|
|
|
by gwern
3292 days ago
|
|
|
> isn't really the average results, but what is possible and the distribution among beneficial results only No, the bad results also matter: you are still spending visitors and revenues in testing out bad variants, which is part of determining the costs and benefits. Even with a bandit approach, you incur logarithmic regret in the number of variants. And testing a bad variant is common: the best category, 'scarcity', has a 16% probability of the variant being harmful. A Value of Information calculation has to take into account the harm done while testing. |
|
|