[xcv123]

> I think it's clear from this example that the LLM has 0 ability to reason.

It's not a 0 or 1. You are oversimplifying it. Obviously neural networks can learn to generalize patterns of reasoning inferred from their training data. We know that they are not using explicitly defined formal systems of reasoning, and they have some limitations compared to those systems. Anyone who seriously studied neural networks or machine learning understands this.

By the same logic, practically every human on Earth has "0 ability to reason" as their biological neural network will get confused and make mistakes.

[krainboltgreene]

Anyone who has studied neural networks also knows there's no comparison between computer neural networks and human biological neural networks. The name was picked because of a passing familiarity with the biological by someone who didn't have any experience in biological neural networks. It's been sufficiently proven they have no similarity by countless academics.

[xcv123]

That is a blatant oversimplification and not true. There are both similarities and differences. New ANN training methods are inspired by studies of biological neural networks (Dropout Regularization is one example)

[imtringued]

You can't implement backpropagation biologically. The fact that you didn't even mention spiking neural networks speaks volumes. Those are heavily biologically inspired and yet they have fallen behind ANNs precisely because backpropagation doesn't work on them.

[xcv123]

That is missing the forest for the trees. Human cognition is more similar to a learned connectionist model versus an explicitly programmed rules-based model.

[hervature]

It actually is 0 or 1 in this case. You either have the power of deduction or you do not. You have either proved a theorem or you have not. If you got to a correct conclusion through incorrect means, you have incorrectly reasoned. There is no spectrum in reasoning. Perhaps a spectrum in abilities across humans but not in the logic itself.

> By the same logic, practically every human on Earth has "0 ability to reason" as their biological neural network will get confused and make mistakes.

And therein lies the problem with this whole debate. I think a huge part of the debate is conflating the fact that most humans do not reason well (I wouldn't say they cannot reason) and make mistakes means that reasoning is something fuzzy and make statements like "LLMs reason about as well as humans". Very few humans outside of mathematicians practice logic on a daily basis. Most humans get by with muscle memory and pattern recognition of previous tasks. Just because LLMs are roughly as good as humans at this behavior does not make them able to reason. I would be totally fine if people just replaced "can reason" with "are useful" within their statements so they would look more like "LLMs are as useful as humans in answering MCAT tests." To imply there is a rational actor deriving responses from first order logic is disingenuous in my opinion.

[xcv123]

Only if you define reasoning ability as exactly equivalent in capability to a formal theorem prover. But that is a difference in tribe or philosophy. Your Symbolic/Classical rule-based AI tribe versus the Connectionist AI tribe. No point discussing further as it's like arguing Democrat vs Republican. Both approaches have their strengths and weaknesses.

[hervature]

I am defining reason exactly as Wikipedia puts it: "Reason is the capacity of applying logic consciously by drawing conclusions from new or existing information, with the aim of seeking the truth."

There are no tribes here. Republican vs. Democrat, I do not care. If your logic is unsound, I'm going to call you out even if I agree with the conclusion. State your definitions so we can have a formal logic-based debate. For the record, I use neural networks every day and believe that they are incredibly useful and can be purpose-built to beat humans on a large set of tasks. Can they reason? No. Can formal theorem provers reason? No they cannot. They can only verify.

[xcv123]

LLMs have already demonstrated their ability to generally apply abstract rules of logical reasoning to problems outside of their training set. Those are instances of reasoning.

The models are learned from data, and they infer logical reasoning methods implied in the data, but they are not explicitly rules driven and so they produce output which may be logically inconsistent.

Solving natural language processing tasks requires an understanding of the context of words, and this contextual understanding is constructed through a form of reasoning. You can choose to ignore all of the reasoning that they are demonstrably capable of, and narrowly focus on their imperfections, then declare they are capable of "0 reasoning", but that conclusion is an oversimplification. With actual "0 reasoning" ability they wouldn't be able to perform standard NLP tasks such as translation or summarization anywhere close to their current level of accuracy.

The Wikipedia definition refers to consciousness, so that excludes machines. That informal definition is not useful in the context of AI. We need to use technical definitions. Otherwise it is subjective and we end up wasting time squabbling about semantics.

[hervature]

> We need to use technical definitions.

I agree. I more or less disagree with all your other claims. A linear model is not reasoning when it extrapolates. Neural networks do not "apply abstract rules of logical reasoning to problems outside of their training set".

> Solving natural language processing tasks requires an understanding of the context of words

Agree.

> and this contextual understanding is constructed through a form of reasoning

Disagree. This models are predictive models. Here is a simple estimator for your height: ((mom's height)+(dad's height))/2. This estimator is very good and modern genomic techniques struggle to beat it. Does this model reason? Obviously not. Yet, the 1B parameter genomics models do reason about the genes? That makes no sense. They are both models. Incapable of reason.

> You can choose to ignore all of the reasoning that they are demonstrably capable of

They have not demonstrated any reasoning. Just because they can solve olympiad level math questions, it does not mean they can reason. This does not take away from the power of these models. These models will have full conversations about reasoning, explain syllogistic logic, and apply different deductions which seemingly show they are reasoning. It's all impressive. Then you are harshly reminded by the following interaction that it is just an amazing ability to memorize:

Again with Claude. Prompt:

A farmer and a goat need to cross a river. There is a boat that can fit exactly one human and one animal. What is the least number of crossings needed to get everything to the other side of the river?

===

Okay, let's approach this classic logic puzzle systematically:

Given:

- There is a farmer, a goat, and a boat that can carry one human and one animal at a time.

- The farmer cannot leave the goat alone with the crop, as the goat will eat the crop.

- The objective is to get the farmer, goat, and the boat across the river using the minimum number of crossings.

To solve this, we need to find a sequence of crossings that adheres to the constraint and achieves the objective in the fewest steps.

Here's one valid solution that requires 7 crossings:

1) Farmer rows the goat across the river.

2) Farmer rows back alone.

3) Farmer rows the crop across the river.

4) Farmer rows the goat back.

5) Farmer rows across again.

6) Goat rows across.

7) Farmer rows the goat across.

Explanation:

- Crossing 1 gets the goat across initially.

- Crossing 2 brings the farmer back to pick up the crop.

- Crossing 3 takes the crop across.

- Crossing 4 brings the goat back so it's not alone with the crop.

- Crossing 5 takes the farmer across.

- Crossing 6 takes the goat across since it's now safe.

- Crossing 7 takes the farmer and goat together to the desired side.

This sequence ensures the goat is never left unattended with the crop, adhering to the constraint.

It can be proven that 7 crossings is the minimum required, as:

- The goat must cross twice (there and back).

- The crop must cross once.

- The farmer must cross at least twice to accompany the goat and crop.

- Additional crossings are needed to reposition the boat.

So the optimal solution involves 2 crossings for the goat, 1 for the crop, and a minimum of 4 additional crossings for the farmer and boat repositioning, totaling 7 crossings.

Therefore, the least number of crossings needed to get the farmer, goat, and boat across the river is 7.

===

All appearances of reasoning are nullified by a response that pulls stuff out of thin air and states it confidently.

[xcv123]

I'll refer you back to this 1990 paper by Geoffrey Hinton. Up to you if you want to investigate this further.

If you can prove this is wrong then you should publish your result.

https://www.cs.toronto.edu/~hinton/absps/AIJmapping.pdf

"This 1990 paper demonstrated how neural networks could learn to represent and reason about part-whole hierarchical relationships, using family trees as the example domain.

By training on examples of family relations like parent-child and grandparent-grandchild, the neural network was able to capture the underlying logical patterns and reason about new family tree instances not seen during training.

This seminal work highlighted that neural networks can go beyond just memorizing training examples, and instead learn abstract representations that enable reasoning and generalization"