[stubish]

I wonder if they actually get meaningful results from MTurk participants; you are getting a very skewed and unnatural demographic, and I don't see how any results based on feedback from MTurk participants can be applied to any normal population. Much more so than the traditional study group of students-who-need-to-pay-rent.

I don't see an ethical problem in how much you pay participants (after all, how much it pays or if it pays at all will be one of the variables the study needs to account for). The ethical problem I see is not caring if the results are meaningful, just publishable.

[mritchie712]

I'd think it's so skewed that it makes the results worthless. I wonder if anyone's done a study on it (e.g. get an actual random US sample vs. a MTurk sample and compare the results).

[phalangion]

> Much more so than the traditional study group of students-who-need-to-pay-rent.

Why do you say that? I imagine the MTurk demographic is much more diverse than "college freshman at X university"

[xienze]

> I wonder if they actually get meaningful results from MTurk participants

I bet they do. One thing not really touched on in the article is that one of the task qualifiers is “acceptance rate”, i.e., how many of your task submissions were accepted by the person that created the tasks. And that number is usually 99.9% acceptance (of _all_ results you’ve ever submitted) or higher. There is a very strong incentive to not get any kind of rejected submission, because it can quickly tank your MTurk profile and shut off lots of opportunities.

[ashelmire]

I know that we use MTurk at the Linguistic Data Consortium (and that others in NLP use it as well). I know that those involved with it put in a lot of work to make sure we get good, useful data - there are many papers and strategies on the subject of crowdsourcing.