[dsawler]

I think "great design can't be tested" is wrong. All design has consequence. If I make an add to cart button on an ecommerce site purple, I can test that to see if it performs better than the red add to cart button. Yes, these decisions (of choosing red or purple) can come from a designer understanding his/her audience, but should never be just taken at the designer's word... it should always be tested to make sure the designed solution is the right one.

[sirclueless]

I disagree: the really important changes take vision and guts, and can only be tested after the fact. Testing things like the color of a call-to-action button can certainly direct you to a local optima, but in a larger sense it misses the point. You can A/B test a whole bunch of small changes, but the really impactful ones take a unified vision that changes the whole design strategy of a website or product. Google didn't go from old Gmail to new Gmail by incrementally changing things based on testing, they took a good hard look at the pain points of the old site and applied a new visual direction to the site. The iPhone didn't come around by tweaking a phone piece by piece until people liked it better. It came around because someone had a vision and saw it through.

Don't get me wrong, usability testing is incredibly important. Every programmer can tell you how much they learned the first time they watched an uncoached newbie try out their software. And even with those paradigm-shifting design changes, you can and should test that the results are really an improvement. But you can't make big important changes by measuring the impact of every little detail.

[SoftwareMaven]

Thank you for putting my thoughts into much better words. The "local maxima" point really nails it.