[equark]

Yes, examining the data will mess up the sampling distribution and invalidate the standard Wald test. But it's absurd in the AB testing context to advocate not acting on your data. Of course it's also absurd to look at conventional p-values if you do. So it's a bit of a Catch-22.

All this confusion goes away if you realize you are interested in p(lift | data) rather than p( data | lift=0). The sampling distribution -- the distribution of the statistic under repeated sampling, p(data | lift=0) -- does not play a role in Bayesian statistics. Obviously the "model" (likelihood/prior) does, but this doesn't include the experimental procedure provided that the experiment is only based on observed data.

AB testing, as a decision procedure, is an area where I don't think the standard frequentist - Bayesian debate applies. The Bayesian decision rule is the only profit maximizing solution. That said, I"m sympathetic to being practical. But all the confusion and conflicting advice related to AB testing stems directly from trying to fit it into a frequentist frame.