[skt5]

This likely means those consuming the outputs of Mechanical Turk don't have a good way to measure the value (aka quality) of the outputs.

If they did - then they shouldn't care whether it's a human or a LLM. And if it's a LLM - then the cost will roughly correlate to the MIN(cost of the LLM, cost of a human) to do the task.

[AndrewOMartin]

I think the "state of the art" of measuring the quality of outputs was to send the same task to multiple "agents" and only accept answers if over a certain amount agree. With some human review and reputation scoring sprinkled on top. It was a while since I was in this field though

[skt5]

This approach does work when there's a clear answer but what about tasks where the correct answer is multi-modal? Incentivizing agreement works only for tasks where there's clear answers.

[renegade-otter]

The problem is bigger. Outside of coding, there is no real way to reinforce a model with pass/fail cycles until it stops hallucinating. This is why customer service uses will always have a problem. This compounds as you chain agents together.

It's like the speed of light - to get to that point, you need exponentially more energy, and you will never ever get there.

[throw310822]

> then they shouldn't care whether it's a human or a LLM.

I imagine that the whole point of posting a task to Mechanical Turk nowadays is that you want it to be completed by humans. Either because you are after the small discrepancy between AI and human performance, or because humans are the object of your investigation.