[msy]

You cannot measure everything. You cannot A/B test frustration, or confusion, or discomfort. You cannot measure slight changes in the way your brand or site is perceived. You can vaguely measure of proxies but they're just that, proxies. The danger of a culture built around worshiping data to the point of ignoring everything else is that you forget that you cannot measure everything, including many of the things that matter most.

[moultano]

>You cannot A/B test frustration, or confusion, or discomfort.

You certainly can. You can measure bounce-rate, abandonment, etc. They are all proxies for the real thing, but are much better than anyone's instincts.

[sokoloff]

Partially true. While you can't directly test for frustration or confusion, what you can test for is the important (to your business) effects of that frustration or confusion.

Higher bounce rates, lower conversion, changes in dominant navigation path, lower average order value, etc. can be determined from A/B or multi-variate testing, and a lot of companies are much less interested in whether a user is confused than they are in whether any potential confusion is causing a reduction in CR% or AOV.

If the article's rant had argued "A/B or MV testing is a hill-climb and incremental changes backed only by test data virtually assures you'll find a local maxima, but only a daring design decision is capable of moving you to a different, higher, hill," I'd be more inclined to agree.

As it stands, it reads to me like a designer who is frustrated by not being able to scratch his own personal itches, without regard to the underlying business results: http://www.artlebedev.com/mandership/140/

[msy]

But if you're measuring via proxies you lose causation. Have those gradual changes that immediately increased conversion or upsell by 2% over the last 3 months cheapened the brand and site, causing long-term damage to its perception in the market place? Has the move to weekly newsletters, that brought that burst of business started to make people think of you as irritating and spammy? A/B testing isn't going to tell you that but those blessed metrics are going to start going south down the road and you aren't going to be able to A/B test your way out of it. Perceptions last, brands are ephemeral and empirical, careful testing indicated New Coke should have been a massive hit.