[jimmar]

I've collected experimental data using Mechanical Turker. In my experience, you have to do a lot of validation to make sure the data you are getting is reasonable. You have to kick people out who aren't paying any attention at all and just want to click through your survey to get paid. (Similar techniques also have to be employed for undergraduate participants who just want to get done with your survey so they can earn their 10 points of extra credit).

If nothing else, Turkers are good at making sure your data collection instruments are working the way that they are supposed to.

[pavel_lishin]

> In my experience, you have to do a lot of validation to make sure the data you are getting is reasonable.

Do you ask any 'trick' questions that are the cognitive equivalent of a captcha?

[jimmar]

I haven't found the need to use "tricks." Basic questions work pretty well. I've done a couple of things. 1) Ask, "Do you have any problems reading English?" Some people with poor English skills, or people being honest, or people not reading carefully enough will click "Yes" thinking that I'm asking them if they speak English. 2) If sound is required, I've played an audio clip and asked people what was playing. They have to be able to identify what was happening from a list of words. This makes sure that they have headphones and can speak English. 3) I sometimes will ask people an obvious question about something that was presented on the previous page to make sure people pay attention. For example, if they have to read an essay on baseball, and the word "baseball" is mentioned 20+ times in a paragraph, I might ask them what sport the essay was about. A surprising number of people miss questions like that. 4) As mentioned by another poster, tell them to select a specific choice (e.g. "Strongly disagree"). I try not to do anything too nitpicky out of respect for people's time and because people aren't perfect.

[VLM]

I've turked for fun and you'd be shocked how many incredibly long psych surveys have scale of 1 to 10 questions like "enter a 5 as the answer to this question if you expect to get paid". Also if a survey asks "how much do you love the new windows phone" you can fully expect the opposite question perhaps on the next page "how much do you hate the new windows phone". Presumably they correlate the data somehow?

I've never run into a serious college professor category of trick question.

Turk is a grind game if you don't need the money. If you're susceptible its going to be a little moth to flame for you and all that matters is your meaningless number goes up, or if you're easily bored like me, you'll turk for like 20 hours one week and then not think about it again, until this HN question, in fact.

I will say that any researcher type would be horribly disappointed by turking because I did it while watching TV, or drinking a beer, and I have a low tolerance not being much of a drinker. Speaking of beer, I am told there is, or was, something of a fad of trying to turk fast enough to pay for your booze. There is no way humanly possible to turk fast enough to drink at a bar, but cheap freshman class beer in a dorm is totally realistic. Can you pay for that six pack on mturk before you finish drinking it? So... I don't think anyone could get homework assignments done on mturk. At least if you want to pass the class or have graduated from grade school. The linked article example is far beyond anything I ever wrote for mturk.

The most insulting and infuriating turk jobs I ran into were not on the author's list, basically being paid to be spammed to. Here's 20 full page variations of a home equity mortgage spam, now pick your favorite one for a nickle. Or call this number and talk to this scammer for a minimum of 5 minutes and then come here and fill out a survey for like 10 cents, at least on the surface its for "quality control" but you know its really to give you the hard sell.

Life is cheap on the turk, and its full of work hits that pay the equivalent of a buck an hour or less, which is eventually super demotivating / depressing. You hear about cool high paying hits but end up filling out 20 page surveys about B.S. for like ten cents, and getting sick of it.

[e12e]

It may be worth noting that making surveys that actually provide good data, perhaps especially longitudinal data (eg: how does Android popularity change over the period 2014-2024), is both science and art. It's also compounded if you try to collect answers in several languages. See eg:

http://www.europeansocialsurvey.org/methodology/improving_qu...