[nuancebydefault]

The thing is, from a written human readable text, there is no single formal reasoning. The text itself is not formal. The facts that kiwis are bigger or smaller might seem irrelevant for counting the amount of kiwis, but there is no formal proof of that possible. I might argue that counting might include volume or weight, you might argue that one kiwi is one kiwi. So saying that llm's don't do formal reasoning is not saying anything, as it doesn't mean anything when you start from written sentences.

My point being, LLMs are capable of reasoning and formal reasoning is meaningless in the context.

[adamc]

Maybe you are right technically, but the fact is that humans can read that text and fairly easily figure out how to reason about it. That's the bar that an agent would need to meet.