[spindritf]

> Same goes for ad critiques. You wouldn't ask "is this a good ad". You'd split it up. "Is this actor better wearing a blue shirt than a red shirt" - ask 10 people.

But you're still asking the "wrong" people. And you're asking, instead of observing the actual behaviour.

[marshallp]

You can have statistics figure that out for you. Are turkers good at judging ads based on past data. If not, maybe it's because you weren't presenting the right metrics for the turkers to judge and so you keep reiterating until you do. This is actually a good business for a startup, you get runaway success such as google has (the data keeps improving your lead over rivals).