[Someone1234]

That most A/B tests don't prove what they aim to prove. People just add a new version of the site, then wait until 100 people have seen it, measure a small improvement and migrate to the new version.

However if you then run the test a second time, but make the "old" site the alternative, you might find it too sees a small measure of improvement and thus you could bounce back and forth between A and B until the end of time.

I am not a statistician so I won't try to give you advice on what a statistically significant result is. However many /many/ articles have been written on that topic, and many products have been designed which you can slot your data into to see if the 'B' option is legitimately better.