Good example of effective marketing. They give you a free, useful plugin, but related plugins are available for a substantial fee. Given that you have to sign up to download the free plugin, they also have a way to propose an upsell in the future (I pretty much expect emails from them).
Yes, visitors get a cookie that has the value of the page they should see on return visits. For instance, if a visitor ends up on variation page #2, they will see variation page #2 from now on. Or at least until they reach the conversion page or the cookie expires in 30 days.
I think it is irresponsible to put tools like this into the hands of non-technical users. The lack of statistical Significance or recommended sample size will cause a lot of people to make uninformed decisions.
That's a more irresponsible stance to take - witholding tools and knowledge because you fear people are ignorant is a sure fire way to keep those people ignorant.
Part of our goal with this plugin is to educate users about these things - look for much more of that coming in the future.
Are you trying to say it is better to have A/B testing without statistical significant as compared no A/B testing at all?
I'd disagree with you completely and since you are taking this stance, I am actually worried now about recommending the plugin. Not caring about statistical significance is equivalent to taking decisions at random. You may better flip a coin, why bother A/B testing? You are letting your users take decisions not knowing if those are real or just random.
Keep in mind that this is just step one of many we'd like to take with this plugin. It's not that we don't care about statistical significance - it just didn't make it into this version, so we'll work to get that in there soon.
We will also be adding more in the FAQ tab that talks about these sorts of issues where users could possibly make decisions too soon instead of letting the tests gather more data.
I am absolutely not suggesting you withhold tools and knowledge, but that you should give them more tools tools that fully inform them. When you show "Improvement: 25%" after say, 20 page views - that is simply inaccurate.
Yes, I noticed lack of statistical significance. If the developer is reading this, I would seriously recommend adding statistical significance. It should be quite easy to implement. I had made on excel sheet for calculating statistical significance so you can use those formulas directly: http://visualwebsiteoptimizer.com/split-testing-blog/ab-test...
That said, I commend the team in bringing easy to use split testing solution for Wordpress!