[michaelt]

> But rather, under a punishing set of Kafkaesque rules, like someone was thinking only of computer programs, oops. "Gamified", with huge negative points penalties and everything. To be under threat of not getting paid at all.

I'm not defending these practices, but to share some context:

One of the problems with getting workers to review ML output is it's incredibly, unbelievably boring. When the task is to review model output you're going to hit the 'approve' button 99% of the time - and when you're being paid for speed, nothing's faster than hitting the approve button.

So understandably a decent number of folks will just zone out, maybe put youtube on in another window, and sit there hitting approve 100% of the time. That's just human nature when dealing with such an incredibly dull task - I know I don't pay attention when I have to do my annual refresher training on how to sit in a chair.

This sort of thing is a big problem for things like airport baggage scanner operators; pilots with their planes on autopilot; lifeguards; casino CCTV operators; and suchlike. There are loads of studies about this kind of stuff.

This makes getting good quality ML output reviews quite tricky. There are ways to do it, though, and you don't have to resort to negative income!

[robertlagrant]

Stack Overflow does this by sometimes prompting you with known bad changes that you shouldn't approve. But then they're managing volunteers, not paying for bad reviews, so they have no money to waste.

[stevage]

I honestly still have trouble believing there are people willing to moderate SO content for free. Do a boring job to make a rich company richer, get paid nothing and occasionally get yelled at.

[Aeolun]

It attracts a lot of petty tyrants, that’s for sure.

[pglevy]

Seems like some of the techniques described here could be part of a larger "accuracy-based commission" form of compensation (as opposed to what is apparently presented).