[btilly]

There are a number of valid ways to run and analyze A/B tests. Bayesian approaches look rather different than what you're describing.

[jfarmer]

The original commenter was using off-the-shelf A/B testing software, so the odds of it doing anything other than a simple t-test are virtually zero. Not sure that the frequentist vs. Bayesean debate is the most relevant thing for him right now.

I felt it best to leave out nuance that didn't help him understand why his software was showing a statistically significant outcome for an A/A test.

[robrenaud]

You are assuming the framework isn't broken or misconfigured.

[jfarmer]

You're not going to out-pedant me, damnit! :P

Given the original comment, yes, I think it's more likely he just didn't understand why a A/A test might sometimes show a false positive.

Even if the system were misconfigured, there's no reason to think it would manifest as a false positive in an A/A test. There are lots of ways it could manifest.