[whyenot]

Suppose in A/B testing A has a higher "conversion rate" than B. Based on the data you collected, you conclude that A is the better option. Sounds good, BUT if you did not take the time to actually do some statistical hypothesis testing, you could be making a mistake.

What are the chances that A and B will have exactly the same conversion rate? For any reasonably large set of data, it is very close to 0. That means that A > B or B > A no matter what A and B actually are. The difference you observed between A and B could be real or it could be noise. Various statistical tests[1] can help you in deciding between the two possibilities.

Flip a coin twice. Lets say you get two heads. If you then conclude that for this coin heads are much more likely than tails, that would be wrong. This is the sort of mistake statistical tests can help you avoid.

[1] I'm being intentionally vague here because what statistical tests you should use depends on what A and B are, what assumptions you are willing to make, sample size, and other factors. A good starting place is probably to use a chi-square test.

[sesqu]

Switching when there is no effect is harmless, from the perspective of the product. It only hurts your takeaways, and if you're doing A/B, you're conceding that your previous takeaways weren't the final word anyway.

Now, if switching is expensive for some reason, and your A/B isn't as conclusive as you'd need, there's a pretty good chance your change resistance will catch that. So even then it's probably not a big deal.