[adwf]

Yeah that's fair enough, but 50% increase in revenue per impression is hardly bad. You can make the argument that it could be better, and I agree with that sentiment within the article. But one of the first things to bear in mind when optimising is to always go for the low-hanging fruit first.

Although it's just my guesstimate, I'm imagine that the broad approach to A/B testing (or just doing it at all in the first place!), will be the most important and easiest fruit to grab. Whilst a multi-armed bandit solution will probably only provide much smaller returns on top. So given a choice between a wikipedia dev concentrating on improving the overall message, as opposed to trying to improve 5 or more cohorts at once - I'd go for the easier option until it stops giving significant returns.

That's not to say it shouldn't be done eventually. Like you said, it's potential money left on the table!