[missingrib]

I think Searle's Chinese Room argument refutes this. LLMs are simply manipulating symbols, they do not have semantic understanding. This is why hallucinations exist. And Searle's argument extends even further than LLMs.

You are basically arguing for a functional account of consciousness, but things like this have been debated for literally decades/centuries in philosophy.

[rerdavies]

Millenia, in fact. The big difference, of course, being that we now have experimental philosophy machines (aka computers). So we can actually put some of these theories to the test, and recognize how utterly inadequate most of the work done on the subject has been. We had a pretty good idea anyway, so it's not a big surprise. Theories of mind have evolved dramatically in the late 20th century. And it's pretty clear that theories of mind will have to be re-done all over again with the advent of LLMs (particularly current-generation LLMs).

The problem with the hallucination argument is (1) that is much less of a problem with good current generation AIs, and (2) living conscious breathing human beings also have a disturbing tendency to make shit up, too. So a tendency to make stuff up doesn't really serve as a disqualifier for consciousness.

Also worth mentioning that the guiding rule of what's philosophical or not is whether it's actually useful. Actually useful philosophy usually becomes something else. Usual some scientific discipline or another. And as it turns out, theories of mind are likely to become extremely useful in the near future. Expect huge advances!

[cauch]

I think one could argue the opposite.

1) Good current generation AIs are specifically trained to reduce hallucinations. If we had new AI system that happened to not have hallucinations as a side effect of their training, then it would be convincing. But here, it looks like we have built a pocket calculator that answer 7+13 = 14, and on top of it, we added a layer that says "if the input is 7+13, then replace the output by 20". This pocket calculator still does not know how to calculate, we just added a layer to hide its mistakes.

2) Not only "make shit up" is not the same as "hallucination" (either "making shit it" is done when the individual knows it is unreliable, or when the individual was given wrong inputs), but the point is not to say "hallucination implies no consciousness", but "large quantities of hallucinations in situations where a conscious system would be unlikely to hallucinate implies no consciousness"

[rerdavies]

Yeah,don't use GPT for that. It really can't do basic arithmetic.

Try Claude, which can.

[cauch]

Wow, I don't think you understood at all.

First, the "13+7" is an analogy. In this analogy, "13+7" is not the real question you ask, it represents _any questions_, not just arithmetic.

But secondly, did you even noticed that in my example, the system answer CORRECTLY "13+7"? So, in my example, the thing I'm talking about and I argue does not "understand" is Claude, even if it is able to answer correctly.

My point is: the "basic LLM" part is creating a mechanism that answer without understanding (as demonstrated for example by ChatGPT failing arithmetic), and the fine-tuning or the harness is just hiding the lack of understanding by adding ad-hoc correction on the residuals. And because it is on the residuals, it looses the logical links (13+7 -> 20 is "logical", it corresponds to the math logic, it corresponds to what you get when you add 13 stones and 7 stones together. The residual is "14 -> 20", which has no meaning in itself)

The ad-hoc correction is either: 1. by training the model so it learns by heart, without understanding, that the symbols "13+7" should lead to "20", 2. or by training the model to use a pocket calculator without understanding arithmetic so it can do it itself.

You can prove that the model does not understand it very simply. Let's take the normal fine-tuned model M1. Now, let's go back to the pre-tuned version, and fine-tune it so it answer "21" to the question "13+7", and use an harness that does "sum(x, y): return x+y+1". This is model M2. M2 will fail to answer "13+7" correctly, it will say "21". And yet, M2 has been trained exactly the same way M1 was. If it is true that the additional tuning "add understanding", M2 will not be possible, it will say "error, error, do not compute, you try to train me to say that 13+7 is 21, but it does not make logical sense to me". But it does not happen: the pre-tuned model has no idea that 13+7=20 is more logical than 13+7=21, and the additional tuning is just helping him returning a more correct answer while still having no idea where this answer comes from.

[VirusNewbie]

>LLMs are simply manipulating symbols, they do not have semantic understanding.

falsify this. Show me a way you'd be able to prove they do/don't, that would work for humans.

[bondarchuk]

Searle's Chinese Room argument is wrong.

[miyoji]

This is not helpful. In what way is it wrong? Does the person in the room know Chinese?

[bondarchuk]

It is a helpful pointer for people who might otherwise assume that a well-known argument by a famous philosopher is sound without checking too deeply. Straightforward refutations can be found on wikipedia or by thinking about it.

[miyoji]

That just isn't true, there are no straightforward refutations of the Chinese Room that are widely accepted. Philosophers disagree about it. It's highly controversial and pretending that it's decided one way or another is not a helpful pointer for anyone.

[bondarchuk]

>That just isn't true, there are no straightforward refutations of the Chinese Room that are widely accepted.

Yes there is, the systems reply is the obvious and correct answer. Philosophers that disagree are simply wrong. In the end what matters is what's true or false, not how many philosophers accept something. You can check for yourself by reading the argument, following its reasoning, and seeing that it is false; and reading the systems reply, following its reasoning, and seeing that it's true (https://plato.stanford.edu/entries/chinese-room/#SystRepl). The case is similar to those mathematical or logical proofs for the existence of god, where obviously fallacious reasoning gets a pass because it confirms deeply held beliefs.

edit: by the way as to your assertion that the argument is controversial and there is no consensus, I just found something funny on wikipedia (https://en.wikipedia.org/wiki/Chinese_room#History):

>Most of the discussion consists of attempts to refute it. "The overwhelming majority", notes Behavioral and Brain Sciences editor Stevan Harnad,[f] "still think that the Chinese Room Argument is dead wrong".[13] The sheer volume of the literature that has grown up around it inspired Pat Hayes to comment that the field of cognitive science ought to be redefined as "the ongoing research program of showing Searle's Chinese Room Argument to be false".[14]

[cauch]

What you are referring to is Searle assertion that "because the Chinese room concept, I conclude that every future human-made systems will be a Chinese room and will never be 'intelligent'".

I think it is an important nuance.

You have to be careful when saying "Searle Chinese room" is dead wrong: the Chinese room concept in itself is useful and not controversial, and it is possible that current LLM are "Chinese rooms", and therefore not 'intelligent'.

[miyoji]

> You can check for yourself by reading the argument, following its reasoning, and seeing that it is false; and reading the systems reply, following its reasoning, and seeing that it's true

You are being tedious. I obviously have done this and I disagree with you. Saying that X is logically true and Y is logically false is not a demonstration of those baseless assertions. This is not helpful, what you're saying isn't true, and what I'm saying is backed up by the wikipedia article. The bit you quote is simply stating that most literature about the Chinese Room is an attempt to refute it, which is obvious, because the people who are convinced see no need to publish saying so. The fact that people keep publishing means that they have not yet succeeded in refuting it.

Or I can simply say this: you've made a mistake in your logic. Actually, the Chinese Room argument is correct. Since you won't explicate your logic, neither will I.

Have a good day.