[bmahmood]

Hi Sean - great point! When I was at Optimizely working on their data science team, we found that on average a test needed 10K-20K unique visitors to reach significance.

We find that this rule of thumb extends similarly to our causal analytics platform. However, we have found that even low-traffic sites are able to get a boost if they are tracking more events on their website (increases the opportunities for signal). Also, our simulations run in minutes on all your historical, rather waiting for weeks for users to be exposed to the test, which speeds up time to insight. If we can not determine significance in our simulation though (due to either sample size or signal), we will designate the projection as a correlation.