[Machow]

"given this measurement and our prior beliefs, the probability of page A being better than page B is X%"

FTFY ;). I think Bayesian methods add a lot of interpretive power, but I'm not sure that it would help people make a correct interpretation. I suspect that if practitioners are neglecting the difference between a one-sided and two-sided test, they will likely forget (or gloss over) what priors are (and their non-trivial implementation).

I definitely agree that their is a disconnect between the math and its interpretation, though.

[jules]

In an A/B test where you usually get so much data, priors honestly don't matter much. Just use a flat prior. You'll overestimate the uncertainty a bit, so you may need a couple more data points than necessary but it's still way less than you'd need for a frequentist method. An A/B testing company could even automatically come up with better priors based on A/B tests that their customers have done in the past.