[0xBA5ED]

That's convenient. But I have a challenge for you if you're brave enough to face your delusions. Paste this into your LLM of choice and see what happens:

"A farmer has 17 sheep. 9 ran away. He then bought enough to double what he had. His neighbor, who had 4 dogs and 14 sheep, gave him one-third of her animals. The farmer sold 5 sheep on Monday and again the next day, which was Wednesday. Each sheep weighs about 150 lbs. How many sheep does the farmer have?"

[tptacek]

17 sheep - 9 ran away = 8 sheep

He bought enough to double what he had: 8 more sheep, so 16 sheep

Neighbor has 4 dogs + 14 sheep = 18 animals

One-third of her animals = 6 animals

But the problem does not say all 6 were sheep. It says “animals.” So the exact sheep count depends on which animals she gave him.

Then:

16 + s sheep from neighbor - 5 - 5 = 6+s

where s is the number of sheep among the 6 animals she gave him.

So the answer is not uniquely determined.

Possible sheep count: 6 to 12 sheep, depending on whether the neighbor gave him 0 to 6 sheep.

(I clipped the GPT5 answer here, but will note additionally that even the LLM built into the Google search results page handles this question; both note the possible trick question with the days of the week.)

[0xBA5ED]

And that's the wrong answer. It's a word problem, not a math problem. Also, if it really was a math problem, it wouldn't be 0-6 sheep from the neighbor, it would be 2-6. So it even failed on the math.

[tptacek]

Are you trying to win this debate with a Facebook "ONLY THE SMARTEST 1% CAN SOLVE" question? The whole point of the question is for some loser to be able to say "no you missed XYZ" ambiguity any time a sane answer is given.

By your logic, the only "correct" answer for an LLM to give to this is "the person who asked you this is fucking with you, this is not a real question". I concede: this is a limitation of modern LLMs: they will try to answer stupid questions.

[0xBA5ED]

No, it's a real question. And if it were a math question. The neighbor has 18 animals, only 4 of which are dogs. The farmer receives 1/3 of those which is 6. So for the farmer to receive 0 sheep would require the farmer to receive 6 dogs. But there are only 4 dogs. LOGICALLY, the farmer must receive at least 2 sheep from the neighbor. There's no ambiguity. That's logic. That's intelligence. It's real actual math. Basic arithmetic. A person can easily sit down and work this out. It illustrates that the AI is generating responses statistically and not actually thinking. There are two full layers of failure here: the word problem, and the math problem underneath it.

[tptacek]

I'm really not interested in this Calvinball argument where we try to conclude whether or not LLMs can do math by avoiding as much as possible actually doing math.

Obviously, they can do math.

[0xBA5ED]

A concise problem that requires actual logic will naturally seem a bit convoluted, but an intelligent being can sit down and work it out logically. Anyway, it's not an argument. It's empirical evidence that supports my argument. You have chosen to ignore it or otherwise rationalize it away. Nothing I can do about that.