[thatthatis]

It's a minor point, and perhaps only a quibble, but the title is wrong.

Wikipedua's AB testing is correct in as far as the problem and approach have been defined. This article suggests an improvement.

Wrong is when they're trying to test for something but due to some error they aren't actually testing for what they think they are.

Something can be accurate/correct but also improvable.

[davidgerard]

That's because this post exists as attempted marketing for Synference.

[feral]

Yes, its marketing, at least in the broad sense of the word - we think people should use contextual bandits more, and we want people to know that.

That's why we're working on this problem.

As I said elsewhere, we think AB-testing is a big improvement over nothing, but, like any powerful tool, it can blind people to its downsides.

Our blog title is designed to point out, succinctly, that we think the vanilla AB-testing framework is wrong for Wikipedia, as an easy-to-understand example. We believe that.

I notice that you volunteer for Wikipedia; I want to be clear, we're not trying to pick on Wikipedia here; I personally love Wikipedia. Its just a good well-known example of a successful AB-testing campaign, that we think could be way improved.

[feral]

Regarding the title, I've updated the title to say 'conceptually wrong' - its more of a mouthful, but maybe what we mean is clearer? I hope our intent was clear in the intro paragraph anyway.

[thatthatis]

I'll take it.