[awhitby]

This is important, because it forces a head-to-head comparison. Otherwise, what knowledge the judge has about the abilities of AI matters a lot.

A well-informed 2023 person talking to ChatGPT will quickly establish that it's not a human, but I'm pretty sure a 1950s person doing the same would swear it was human, if an odd human, because they couldn't conceive of a computer being that fluent. But force them into this head-to-head scenario and they will make the right choice.

[chongli]

In the head-to-head comparison, what is the goal of the human talking to the judge? Are they trying to fool the judge into thinking they’re the AI? Or are they trying to make it easier for the judge to spot the AI?

Because this really matters. If you put me up against ChatGPT I’m going to talk briefly and informally, with lower case text. I’ll answer their questions succinctly without much elaboration. And I think most judges would then easily spot ChatGPT due to its formality and perfect grammar.

On the other hand, there are chat bots designed to pass the Turing test which speak informally and even ignore questions. They tend to fool people into thinking they’re a bored teenager.

[SAI_Peregrinus]

> Or are they trying to make it easier for the judge to spot the AI?

The latter. Both A and B (human and AI subjects) are trying to convince the judge that they're the human.