[devit]

The "person one" vs "person two" bias seems trivially solvable by running each pair evaluation twice with each possible labelling and the averaging the scores.

Although of course that behavior may be a signal that the model is sort of guessing randomly rather than actually producing a signal.

[harrisonjackson]

Agreed on the second part. Correcting for bias this way might average out the scores but not in a way that correctly evaluates the HN comments.

The LLM isn't performing the desired task.

It sounds possible to cancel out the comments where reversing the labels swaps the outcome because of bias. That will leave the more "extreme" HN comments that it consistently scored regardless of the label. But that may not solve for the intended task still.

[rahimnathwani]

  The LLM isn't performing the desired task.

It's 'not performing the task', in the same way that the humans ranking voice attractiveness are 'not performing the task'.

I wouldn't treat the output as complete garbage, just because it's somewhat biased by an irrelevant signal.