[em441]

Hi! This is Emily, the stats brains behind this. My research indicated that 80% significance was commonly used for A/B Testing (and of course any significance level >80% would be even more conservative). I definitely see your point of the advantage of a continuous test and I think for the expert being able to see the exact significance level for the test would be very useful. However, our thoughts were that the average user might not necessarily have the expertise to know what a "good enough" level of significance would be for any test. Rather than having to educate every user on what significance means and how to interpret it we decided that a "yes, significant" or "no, not significant" would be more easily interpreted by all, regardless of their statistical background. If there is demand for a more continuous approach, it certainly could easily be implemented.

[tedsanders]

Thanks for the reply. I guess it's a good reminder for everyone building statistical products (or even non-statistical products) that the goal is not an optimal formula, but a result that makes users comfortable.