[knuthsat]

A/B tests work fine if the signal you are measuring is strong. This is not the case here.

Is it even fine to use the distribution assumptions in the later analysis?

Looks like these assumptions combined with a higher conversion rate on day 2 for control is the main reason for the surprising result (control is obviously spread out).

[jefftk]

> A/B tests work fine if the signal you are measuring is strong. This is not the case here.

The (fictitious) signal they are discussing here is very strong. Scroll down to the figure labeled "posterior distribution of p" and you can see that the two distributions barely overlap.

[knuthsat]

Yes, I saw the figure and that's why I commented that the day 2 conversions for the control are basically giving all of the information in the assumed model.

To me it just looks like a whole new batch of assumptions. Might be fictitiously valid or not.