[gweinberg]

I don't know why you are getting downvoted. I was disappointed the paper didn't give examples of the LLM confidently expressing its incorrect reasoning.

From a human perspective, I think there are three ways to get the answer wrong: failure to realize that Alice's brother has pretty much the same number of sisters as Alice herself, failure to realize that the brother has one additional sister, namely Alice, and failure to successfully add one to the number of sisters. But that assumes that the LLM is more or less reasoning like a human. It may well be "reasoning" more along the lines of "I've seen lots of story problems like this, the modal answer was five, I'll say five"