[altmanaltman]

But wouldn't every one of those multiple answers be the correct one in this case? Like it can say child a or child b or child c (hypothetical) and while there are mutiple answers, each of them is a logically right one for the question "Who is her child?" no? So how do we judge what is the absolute right answer to that? its ambigious when you say child

[niam]

Zooming out to the original complaint that "A is B" doesn't imply "B is A" in common English, and then further -- to the goal of having an LLM predict tokens that map closely to truth/logic/helpfulness:

I don't think a person speaking plain English in most contexts should be seen as "correct" to answer the question with a non-list answer, even if the question is shaped to expect one, unless there's an established confidence that the shape of the question wasn't made in error.

If someone asked me in real life who "my child" is on stage, and I had multiple children on stage, I would first say that I had multiple children there, rather than choosing one from the set. It would be most helpful for an LLM in my position to do the same, rather than infer that [because Timmy is niam's child, niam's child ought to be Timmy when queried].

[altmanaltman]

Yeah I guess that makes sense and which I was getting at. Even though logically it a non-list answer to that might be "correct" but not as helpful as returning the list answer and clarifying that there are multiple children. And I guess the later is also kind of more intelligent if we think about it even though it doesn't fully confirm to the exact prompt.