[imx]

Mturk manuals are junk and very hard to follow. HIT data cleansing is the biggest issue. Instead of using the command line tool, use their API to integrate into your app, as it will save plenty of time down the road...

To "weed out" ineligible workers, try this approach: 1. Post a bunch (1000-5000) of cheap multiple-choice HITs. 2. Allow no more than 10 hits per worker. 3. Each hit to get 3 responses from different workers. 4. Review answers, compile the list of "good" workers, blacklist the "bad" ones. 5. Post another bunch of HITS, make them available for eligible workers only (found in step 4), this time the HITs might be more demanding, individually review results for each worker -> the best ones go on your "preferred worker" list. 6. Repeat steps 1-5 as necessary.

From then on it's fairly safe to rely on mturk workers from your preferred list.

[user24]

looks like there's a need for a mturk preferred worker aggregation service.

[patio11]

Seriously, for being a fire-and-forget API to the lowest possible level of human tasks, it requires a heck of a lot of hands on management, including arguing with identifiable people over two cents. (YOU DIDN'T SAY TO TURN OFF CAPS. I wish I were exaggerating.)

I ended up writing off five hours to goodwill when I did a project with a $100 turking component for a client. To use a line favored by my old Indian colleagues: if you pay peanuts, you get monkeys. Lesson learned.

Next time I will just find a freelancer with a high tolerance for repetition.

[todayiamme]

>>>Next time I will just find a freelancer with a high tolerance for repetition.<<<

Can you shoot me an email when you do?

I have the ominous feeling that it will be mind numbingly boring, but nonetheless money is money.

[davidu]

it's called crowdflower.

[wizard_2]

These guys constantly prove they know how to manage the kinds of problems Turk and other sources bring to the table.

[pault]

Until someone uses mturk to spam your ratings.

[barefoot]

Does this actually happen?