[Tinyyy]

I think there can be prior that a professional looking ad can generate more clicks. Your argument shows a lack of statistical understanding - conditional on this data, the Bayesian approach would be to update the prior (whether A is better, or they’re equally as good) with the data collected. With such a small dataset, you might end up with a belief that there’s a 60% probability that B is better than A, but that’s not significant enough to conclude that B is in fact better than A, as you still have a lot of uncertainty.

With a prior that A is superior, you may still end up believing that A > B after updating, because there’s just so little data.

[bpodgursky]

I addressed in my second sentence that I disagree with that prior. I understand the statistics perfectly well.

And my main point is that a 60% probability is in fact actionable in the real world, in a situation where you are forced to take action with incomplete information. Assuming you are running an ad campaign, you have to choose one of the two.

P=.95 still is an arbitrary threshold, even if it's a commonly used one.

[mlyle]

Yes, but the significance is high here; it's a pretty (un)lucky outcome to get if A and B are equivalent, let alone if A is better than B.