[rlt]

Is this with the “temperature” parameter set to 0? Most LLM chatbots set it to something higher.

It would be interesting to try varying it, as well as the seed.

[JoelEinbinder]

Temperature doesn't play a role here, because the LLM is not being sampled (other than to generate the candidate answers). Instead the answer the llm picks is decided by computing the complexity for the full prompt + answer string.