[rxin]

This is just very different mentality between thinking as a designer vs thinking as an engineer.

Google in particular is a data driven company. Most decisions are made by results from experiments. Google is able to do this because the economies of scale. They have tons of data. As a matter of fact, the "experimental traffic" is higher than 100%, indicating multiple levels of experiments are conducted in every search (on average). I remember the quote from a session "A/B testing is for marketers. Engineers run multivariate experiments."

In terms of traditional designers: their process is more of an art than science. They think, and trust their instincts. They traditionally operate in a setting that can't afford this kind of experiments.

The reality is people are weird and unpredictable. It's statistics and hard science when it's backed by data.

The challenge, of course, is that it is easy to iterate incrementally using measurement data. But for disruptive changes, although you could measure it, the change itself (the alternatives) is often not obvious.

[msy]

You cannot measure everything. You cannot A/B test frustration, or confusion, or discomfort. You cannot measure slight changes in the way your brand or site is perceived. You can vaguely measure of proxies but they're just that, proxies. The danger of a culture built around worshiping data to the point of ignoring everything else is that you forget that you cannot measure everything, including many of the things that matter most.

[moultano]

>You cannot A/B test frustration, or confusion, or discomfort.

You certainly can. You can measure bounce-rate, abandonment, etc. They are all proxies for the real thing, but are much better than anyone's instincts.

[sokoloff]

Partially true. While you can't directly test for frustration or confusion, what you can test for is the important (to your business) effects of that frustration or confusion.

Higher bounce rates, lower conversion, changes in dominant navigation path, lower average order value, etc. can be determined from A/B or multi-variate testing, and a lot of companies are much less interested in whether a user is confused than they are in whether any potential confusion is causing a reduction in CR% or AOV.

If the article's rant had argued "A/B or MV testing is a hill-climb and incremental changes backed only by test data virtually assures you'll find a local maxima, but only a daring design decision is capable of moving you to a different, higher, hill," I'd be more inclined to agree.

As it stands, it reads to me like a designer who is frustrated by not being able to scratch his own personal itches, without regard to the underlying business results: http://www.artlebedev.com/mandership/140/

[msy]

But if you're measuring via proxies you lose causation. Have those gradual changes that immediately increased conversion or upsell by 2% over the last 3 months cheapened the brand and site, causing long-term damage to its perception in the market place? Has the move to weekly newsletters, that brought that burst of business started to make people think of you as irritating and spammy? A/B testing isn't going to tell you that but those blessed metrics are going to start going south down the road and you aren't going to be able to A/B test your way out of it. Perceptions last, brands are ephemeral and empirical, careful testing indicated New Coke should have been a massive hit.