[probably_wrong]

I had to do some digging into crowdsourcing for my PhD thesis, and these numbers are well reported in the literature.

Our group made the calculations to ensure that our task would pay at least minimum wage. As a result, we had to throttle our participants because they would hit our servers pretty hard.

That's not to say that the experience was easy - we had to implement every possible sanity check, and even then we ended up throwing away half our data. For us it was still worth it, since twice the rate was still cheap and we knew not to trust the internet. But a less internet-savy researcher blindly trusting their data would have gotten some surprising results.

[namanyayg]

What kind of sanity checks did you implement, or how did you know what data to throw?

[probably_wrong]

We had an interactive task (hosted in our own server) and a questionnaire afterwards. Sanity checks off the top of my head:

* Control words: our server gave them two (unique) control words (one for joining, one for winning), and we asked for them in the post-task questionnaire. Some of these words ended up being reused among several participants, even from participants that never even started the task.

* IP checks: we had an experiment that you could "win" (and earn a bonus), but you only were allowed to play once. Some people restarted the task several times, so we only used the first attempt in a sequence (as reported by their IP and timestamp).

* Data thrown away: we further removed data where we had more than one player per IP (to control for both multiple accounts per person and use of proxies), experiments that were way too fast, and experiments with unsupported browsers (which we explictly mentioned in the description).

Regarding money, we were not allowed by the TOS to withold payment to anyone that filled the questionnaire, even if we knew they did it in bad faith. We therefore implemented the "winning" bonus, and also gave bonus to people who lost but really tried.

I want to point out that a LARGE percentage of participants played honestly, and some of their data was thrown away only out of an abundance of caution. Once you keep the first bad apples out, they simply move to other, easily-exploitable tasks.

[bob_theslob646]

>Our group made the calculations to ensure that our task would pay at least minimum wage. As a result, we had to throttle our participants because they would hit our servers pretty hard.

What do you mean by this? Do you mean purely requests per second or what?

[probably_wrong]

Yes, requests per second. Everyone would jump to our task. Compared to "regular" experiments, where getting participants requires active recruiting, this was completely unexpected.