|
|
|
|
|
|
by leo_pekelis2
4161 days ago
|
|
|
Glad you like that we’re changing things up! You do bring up a good point. Even though a sequential test is able to be called much earlier than a fixed horizon test (note this only happens when the effect size is large enough to still guarantee Type I error control), it does not change the fact that estimates of the effect size are more variable when there are fewer visitors. The way we are addressing this is to make confidence intervals more prevalent in our platform. The width of confidence intervals represents our uncertainty in the magnitude of the true effect size with the information currently available. They correctly get more narrow as the experiment goes on as there is increased information from more visitors. |
|
|