[Yizahi]

No they don't. When queried how exactly did a program arrive to a specific output it will happily produce some output resembling thinking and having all the required human-like terminology. The problem is that it doesn't match at all with how the LLM program calculated output in reality. So the "thinking" steps are just a more of the generated BS, to fool us more.

One point to think about - an entity being tested for intelligence/thinking/etc only needs to fail once, o prove that it is not thinking. While the reverse applies too - to prove that a program is thinking it must be done in 100% of tests, or the result is failure. And we all know many cases when LLMs are clearly not thinking, just like in my example above. So the case is rather clear for the current gen of LLMs.

[roxolotl]

This is an interesting point but while I agree with the article, don’t think LLMs are more than sophisticated autocomplete, and believe there’s way more to human intelligence than matrix multiplication humans also cannot explain in many cases why they did what they did.

Of course the most famous and clear example are the split brain experiments which show post hoc rationalization[0].

And then there’s the Libet experiments[1] showing that your conscious experience is only realized after the triggering brain activity. While it’s not showing you cannot explain why it does seem to indicate your explanation is post hoc.

0: https://www.neuroscienceof.com/human-nature-blog/decision-ma...

1: https://www.informationphilosopher.com/freedom/libet_experim...

[Yizahi]

I agree, but here we are veering into more complex decision making. I was talking about much simpler cases, like for example going through a handful of simple steeps for simple task. For example addition, ask a person to sum two numbers and then ask to explain what he just did step by step and a person would be able to do it. Person may even make a mistake in process but the general algorithm will be matching what actually happened. Query LLM for the same, and while LLM answer will be correct for a human, it won't match what LLM actually did to calculate. This is what outs LLM "thinking" for me, they just generate a very plausible intermediate steps too.