Know a guy who's iPhone app uses MT to do pseudo-automated video image/motion recognition & processing. He had to train & retain his Turks, who seem quite content doing the work as an ongoing project. (Not sure if he wants further details revealed.) Between this and the article, seems the untold story may be that blind/random "turking" may be frustrating & unprofitable, but finding & developing a relationship on a project can work well for all involved.
I've used MTurk for a very subjective task: "Value the object in the image" for which I paid $0.10 per valuation (a maximum of 30 seconds was spend on the task).
The valuations were fair but the workers of Mechanical Turk may have a different view of how much something "home made" or "retro" may be worth (typically far lower than the object would be worth for someone in the market for said object).
I worked for years on systems built on top of MTurk (building trust systems, scaling tasks, etc)
The hardest thing is always task design, as the author discusses. It's quite challenging to create a task that will work in all browsers, across cultural and language boundaries, and to pay enough to incentivize the worker (while simultaneously little enough to make all the effort worthwhile).
The final effect is a bit dehumanizing on both ends. The workers feel alienated and mistrusted by the people requesting the tasks. The people requesting the tasks feel that the workers are lazy / "scammers" trying to cheat them.
We don't fully even know how to think about this kind of work yet - ideas like "hourly wage" don't seem to fit for someone working completely independently, at their own pace, possibly distracted or impaired.
Still, despite the trouble, it seems like one of the most promising fields for interaction design. There's the possibility to really improve peoples' lives (or make them worse).
i've used it a couple times in a lame attempt to find leads. IIRC, I paid $.40 to find a name/phone number/website of construction equipment rental companies in 50 different cities. They did exactly what I wanted them to do, although I never did anything with the results. One guy seemed to pick up most of the jobs, so I tipped him an extra $10.
I've collected experimental data using Mechanical Turker. In my experience, you have to do a lot of validation to make sure the data you are getting is reasonable. You have to kick people out who aren't paying any attention at all and just want to click through your survey to get paid. (Similar techniques also have to be employed for undergraduate participants who just want to get done with your survey so they can earn their 10 points of extra credit).
If nothing else, Turkers are good at making sure your data collection instruments are working the way that they are supposed to.
I haven't found the need to use "tricks." Basic questions work pretty well. I've done a couple of things. 1) Ask, "Do you have any problems reading English?" Some people with poor English skills, or people being honest, or people not reading carefully enough will click "Yes" thinking that I'm asking them if they speak English. 2) If sound is required, I've played an audio clip and asked people what was playing. They have to be able to identify what was happening from a list of words. This makes sure that they have headphones and can speak English. 3) I sometimes will ask people an obvious question about something that was presented on the previous page to make sure people pay attention. For example, if they have to read an essay on baseball, and the word "baseball" is mentioned 20+ times in a paragraph, I might ask them what sport the essay was about. A surprising number of people miss questions like that. 4) As mentioned by another poster, tell them to select a specific choice (e.g. "Strongly disagree"). I try not to do anything too nitpicky out of respect for people's time and because people aren't perfect.
I've turked for fun and you'd be shocked how many incredibly long psych surveys have scale of 1 to 10 questions like "enter a 5 as the answer to this question if you expect to get paid". Also if a survey asks "how much do you love the new windows phone" you can fully expect the opposite question perhaps on the next page "how much do you hate the new windows phone". Presumably they correlate the data somehow?
I've never run into a serious college professor category of trick question.
Turk is a grind game if you don't need the money. If you're susceptible its going to be a little moth to flame for you and all that matters is your meaningless number goes up, or if you're easily bored like me, you'll turk for like 20 hours one week and then not think about it again, until this HN question, in fact.
I will say that any researcher type would be horribly disappointed by turking because I did it while watching TV, or drinking a beer, and I have a low tolerance not being much of a drinker. Speaking of beer, I am told there is, or was, something of a fad of trying to turk fast enough to pay for your booze. There is no way humanly possible to turk fast enough to drink at a bar, but cheap freshman class beer in a dorm is totally realistic. Can you pay for that six pack on mturk before you finish drinking it? So... I don't think anyone could get homework assignments done on mturk. At least if you want to pass the class or have graduated from grade school. The linked article example is far beyond anything I ever wrote for mturk.
The most insulting and infuriating turk jobs I ran into were not on the author's list, basically being paid to be spammed to. Here's 20 full page variations of a home equity mortgage spam, now pick your favorite one for a nickle. Or call this number and talk to this scammer for a minimum of 5 minutes and then come here and fill out a survey for like 10 cents, at least on the surface its for "quality control" but you know its really to give you the hard sell.
Life is cheap on the turk, and its full of work hits that pay the equivalent of a buck an hour or less, which is eventually super demotivating / depressing. You hear about cool high paying hits but end up filling out 20 page surveys about B.S. for like ten cents, and getting sick of it.
It may be worth noting that making surveys that actually provide good data, perhaps especially longitudinal data (eg: how does Android popularity change over the period 2014-2024), is both science and art. It's also compounded if you try to collect answers in several languages. See eg: