[btilly]

I would also suggest switching from chi-square to the g-test.

If you look at the history, I believe that you'll find that Pearson originally came up with the g-test as an approximation to an exact test, and then found the chi-square as an easier to compute alternative. This mattered back in the days of pencil and paper, but there is no excuse today to use the worse technique.

I'm going to avoid long discussions about the advisability of taking multiple looks at results with a classical statistical test. But see my incomplete series at http://elem.com/~btilly/ab-testing-multiple-looks/index.html for some of the considerations.

I never got into Bayesian statistics in there. In general they depend on the existence of a prior distribution. Careful treatments will talk about this. Sloppy ones assume one, don't talk about the one that they assume, and then quote results without letting you know about this important assumption. As long as you accept that assumption, they work well. But sometimes can be confusing to explain. (Until people "get" it. Then it can become irritating getting them to STOP explaining it!)

If you want to discuss these issues more, my email is my name at gmail.com.

[em441]

"Upgrading" to the g-test is certainly something we could implement in the future - Pearson's Chi Square was simply a starting place. I will definitely have a look over the link you provided. Thanks!