| You don't ask assembly line workers to build an amazing car on their own in a single step. Similarly, you shouldn't ask low-paid information workers to synthesize amazing text on their own in a single step. I think that your HIT design highlights several common mistakes requesters make on MTurk: - You are underpaying for the task (would you write a good review of Berkeley, CA for $1 for a stranger?) - You provide no aggregation or verification step, to ensure that turkers know their work should jive with other turkers' output. You also give no indication that such verification is possible or likely to happen. - Your task output is poorly defined and open to interpretation. You may have asked a straightforward question, but I assume you placed a blank textbox on the screen and expected well-formed paragraphs in return. If you want a great example of text synthesis of relatively high quality using MTurk for prices in the range of your budget, see http://borismus.com/crowdforge/ If you want to learn more about how to design HIT workflows, see http://projects.csail.mit.edu/soylent/ (disclosure: I share an office with and work with Michael Bernstein, but not on this work). One of Soylent's contributions was the Find-Fix-Verify design pattern, which helps with some of the problems you raise. Your task is even harder, of course, since you require subject-matter experts in a fictional location. So perhaps MTurk is the wrong crowd for your task. |