[olavolav]

Regarding that last bit: You will clearly get better results by using A/B tests than by having humans take a guess as to what works best – but as you already pointed out, those are not the only two possibilities. Different varieties of mutli-armed/contextual/etc. bandits, and then the new wave of reinforcement learning tricks (i.e. AI) are much better suited to the problem.

(Full disclosure: I didn't write the article, but I am the CTO of the company that the article is about.)