That's the worst. For mature products, all that A/B testing does is test how close the old UI is to the new UI. Users are scared of change, to the point that when Ebay wanted to change the color of their banner, changing the color instantly was a travesty to their users so instead, Ebay transitioned the color slowly, over months, to yellow.[0]
[0] https://theuxblog.com/blog/redesigning-user-experience