[squarecog]

Co-author of the Twitter post here.

I don't necessarily disagree with you! Though I hope you are not calling Twitter a fly-by-night marketing startup :).

As a company, we've found A/B testing very valuable, but it certainly has limitations, and pitfalls that one must be fairly experienced to avoid. You can see some descriptions of what we do to achieve that in other posts in that series. But it's fair to say that this takes some work (like the work described in the "second control" post) to avoid badness, and it's fair to question if other methods can get us to the same end result -- ability to evaluate impact of proposed product changes -- cheaper, or more accurately.

[p4wnc6]

> Though I hope you are not calling Twitter a fly-by-night marketing startup :)

That remains to be seen.