[bredren]

Yes, this is not labeling what is an apple and what is a pear. "Annotation" does not do this work justice.

For a coding agent, for example, there is *very detailed* analysis of the turns and ranking of different portions of the conversation.

Adherence or deviation from specific rules matters. Writing quality matters. Expertise in the topic under discussion matters. Having intuition for the tone and beat of a good conversation matters.

Scoring a 15-20 turn conversation can easily take two and a half hours.

Clicking submit does not mean the author is done. Many annotations will be turned back to them by a reviewer to touch up in some way.

This work can be far more mentally taxing than programming, is measured much more by completions more of a timed exercise than SWE.

FWIW, Meta employees would probably make great coding agent conversation annotators. But it is absolutely not SWE and they won't enjoy it (for long).