The space of possible answers is more than one is all GP is saying. Mary Lee Pfeiffer had 1 son, Tom Cruise, and 3 daughters. That's why saying Who is Mary Lee Pfeiffer's son was an identity, because it strictly identifies (or singles out) Mary's son, Tom.
Kind of a weird way to draw an analogy, but in math it's kind of like |x|=2 (the absolute value of x is 2) the answer for the value of x is -2 and 2 sure you could reply that the answer is 2 and be correct (even though you would still be missing something, because the space of possible answers includes both 2 and -2). To relay that back to Mary Lee Pfeiffer saying she has Tom Cruise as a child is correct, but the actual answer could include any 4 of her children (including Tom or one of the 3 daughters) and still be correct.
Yeah i understand that but it is logically right to reply with "Tom Cruise" or any of the girls to that question because by its structure it requires only 1 of the 4 answers since it asks for a singular child right?
Or is it like we are saying while that is logically correct, its not the actual answer and the model should reply "they have more than one child, here is the list of children" and that would be a more accurate one even though the prompt strictly asked for just 1 child?
> Yeah i understand that but it is logically right to reply with "Tom Cruise" or any of the girls to that question because by its structure it requires only 1 of the 4 answers since it asks for a singular child right?
Yes that is correct it is logically correct to reply Tom Cruise or any of the girls, and that was their point that there are four possible answers to one question.
> Why would it have several answer when you are asking for a singular child?
By the way this quote was the focus of my GP comment since I didn't quote it there.
> Or is it like we are saying while that is logically correct, its not the actual answer and the model should reply "they have more than one child, here is the list of children" and that would be a more accurate one even though the prompt strictly asked for just 1 child?
This was not his point so I feel like we are moving the goal posts a bit, so no though that could have been what's said I don't think that's what was really being said.
But wouldn't every one of those multiple answers be the correct one in this case? Like it can say child a or child b or child c (hypothetical) and while there are mutiple answers, each of them is a logically right one for the question "Who is her child?" no? So how do we judge what is the absolute right answer to that? its ambigious when you say child
Zooming out to the original complaint that "A is B" doesn't imply "B is A" in common English, and then further -- to the goal of having an LLM predict tokens that map closely to truth/logic/helpfulness:
I don't think a person speaking plain English in most contexts should be seen as "correct" to answer the question with a non-list answer, even if the question is shaped to expect one, unless there's an established confidence that the shape of the question wasn't made in error.
If someone asked me in real life who "my child" is on stage, and I had multiple children on stage, I would first say that I had multiple children there, rather than choosing one from the set. It would be most helpful for an LLM in my position to do the same, rather than infer that [because Timmy is niam's child, niam's child ought to be Timmy when queried].
Kind of a weird way to draw an analogy, but in math it's kind of like |x|=2 (the absolute value of x is 2) the answer for the value of x is -2 and 2 sure you could reply that the answer is 2 and be correct (even though you would still be missing something, because the space of possible answers includes both 2 and -2). To relay that back to Mary Lee Pfeiffer saying she has Tom Cruise as a child is correct, but the actual answer could include any 4 of her children (including Tom or one of the 3 daughters) and still be correct.