| It’s unclear to me that highly trained professionals are any better at this than a/b tests. Usually it’s the professionals themselves telling us that they’re better. I guess it depends on where any individual places the task on the skill/luck or simple/complex scale. If the task (in this case, a design) is very simple — how many users who want to complete their voter registration form are able to — then experts may have an advantage. OTOH, if it’s a complex space like facebooks news feed, I don’t think experts are the way to go. They simply think that they are. FWIW, I’ve run hundreds of multivariate tests in my career. Every place I go, the design team is upset by the process, for the same reasons argued here. Every time, I challenge them to come up with 1 winning variation and 3 losers. If the designer-identified “winner” performs best, i pay them $10. If any of the losers wins, I get $10. I have yet to meet a designer that can, a priori, pick the winning variation better than random chance would suggest. Expertise is not valuable in an unpredictable context. Complex user facing systems like the news feed are impossible to predict. I’m channeling Kahneman and Mauboussin here, but their theory explains my experience very well. |
Note that talking to customers is not easy too. To the point that you need to psychoanalyze all the feedback to understand why they make those comments. Useful feedback is very rare. Customers are just not usability experts, they don't really know what they are talking about.