[maeil]

> "An AI assistant was tasked with {task}. The relevant information for their task was {context}. Their answer is {answer}. The correct answer should be something like {ground truth}. Is their answer correct?"

If you have a ground truth, what was the purpose of asking the AI assistant for an answer in the first place?

[senko]

When you're writing a test, you usually know the correct answer for that specific combination of input parameters.

[maeil]

Looking back at it, I must have been very tired when I wrote that!

Or maybe I was thinking about cases where the ground truth is difficult to establish.

[pletnes]

The cat is dead. The cat is no longer alive. These are equivalent enough, usually, but fails string comparison.

[ghm2180]

Or like of you did ai voice calling the goal was XYZ did the conversation get, in so many words, to XYZ?