[thomasahle]

> only whether the participant can guess the (possibly cherry-picked) answer the author decided on

My understanding is that the quiz samples a new GPT-4 answer every time you use it. That's why you put a confidence rather than a 0%/100% answer. There's always a chance it'll fail by freak accident.

[Sophira]

If you're basing this on the animation used when revealing the answer, that's a fake effect. The source code[0] reveals that there's a typewriter effect that plays out when you select to answer the question.

Also, the commentary on the answers refers to specific parts of the answers. For it to be as in-depth as it is, it would have to be either pre-written or the commentary also generated by GPT on the fly. (And of course it wouldn't make sense to do that given the nature of the quiz.)

[0] https://nicholas.carlini.com/writing/llm-forecast/static/que...