[gwern]

Those answers are bizarre. Use a t-test or z-test?! That is both less interpretable* and less efficient than doing a test of difference in proportions (based on a chi-squared) or better yet, doing a generalized linear regression with a binomial link which lets you test multiple interventions and include covariates and other things. These are little harder than using a t-test and work much better.

(Of course, even better would be a Bayesian decision analysis which takes into account the full posterior distribution, costs, and benefits, allowing one to decide whether the evidence is enough to justify the more expensive box or whether one should extend the experiment further to collect more data.)

* quick, if conversions are 0.25%, what does a 0.1SD mean difference between arms of an A/B test mean? 'Uh...' Exactly.

[jupiter90000]

To be fair, a couple answers there mentioned the Chi-squared approach, and a z-test could also be appropriate for testing a difference in proportions. I think your Bayesian approach would make the most sense for this kind of stuff, to examine the posterior especially. There's other things that could be done than what you mentioned too to assess a difference. I'm not convinced the parametric assumptions made by the standard tests mentioned (including difference in proportions, whether a z-test or Chi-square test) would hold up well for this kind of data, it would take further examination. I think a z-test or Chi-square test are a decent starting point, and could be fine to use if the assumptions made for those tests are met (to defend some of the Quora answers). The issue is often tests are used without folks either knowing how to check or ignoring the underlying assumptions, but that isn't an issue with the test itself I'd argue.

[gwern]

> I'm not convinced the parametric assumptions made by the standard tests mentioned (including difference in proportions, whether a z-test or Chi-square test) would hold up well for this kind of data, it would take further examination.

I'm not sure what you mean. If all you have is binary data, there's not much you can do but a chi-squared test or binomial regression. As I understand it, a chi-squared is considered nonparametric: https://en.wikipedia.org/wiki/Chi-squared_test https://en.wikipedia.org/wiki/Fisher's_exact_test

[jupiter90000]

Yes, you're right that chi-squared is non-parametric, my mistake. However, even non-parametric tests can have assumptions that should probably be checked into at least for a given data set. It may be robust to the violations the data collected may exhibit, I'm not sure. This article lists the assumptions for chi-squared test: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900058/