The report states humans take ~60s per answer and GPT-4 takes ~3.0s, thus just 20 times time / 7 times cheaper is the baseline for the highest quality result available.
Faster models are 2-3 times faster, but 3s is already enough even for real-time application
The cost, hovewer, goes down to 1/1000 for cheaper models (and they can still 100% some of the datasets), meaning that for the same price as a human annotator you could have 1000 parallel realtime LLM annotators
The cost, hovewer, goes down to 1/1000 for cheaper models (and they can still 100% some of the datasets), meaning that for the same price as a human annotator you could have 1000 parallel realtime LLM annotators