[siglesias]

"Actual understanding" means you have a grounding for the word down to conscious experience and you have a sense of certainty about its associations. I don't understand "sweetness" because I competently use the word "sweet." I understand sweetness because I have a sense of it all the way down to the experience of sweetness AND the natural positive associations and feelings I have with it. There HAS to be some distinction between understanding all the way down to sensation and a competent or convincing deployment of that symbol without those sensations. If we think about how we "train" AI to "understand" sweetness, we're basically telling it when and when not to use that symbol in the context of other symbols (or visual inputs). We don't do this when we teach a child that word. The child has an inner experience he can associate with other tastes.

[bonobo]

You mentioned experience, but it's not clear to me if you mean that it's a requirement for "actual understanding." Is this what you're saying? If so, does that mean a male gynecologist doesn't have an "actual understanding" of menstrual cycles and menopause?

I think about astronomers and the things they know about stars that are impossible to experience even from afar, like sizes and temperatures. No one has ever seen a black hole with their own eyes, but they read a lot about it, collected data, made calculations, and now they can have meaningful discussions with their peers and come to new conclusions from "processing and correlating" new data with all this information in their minds. That's "actual understanding" to me.

One could say they are experiencing this information exchange, but I'd argue we can say the same about the translator in the chinese room. He does not have the same understanding of chinese as us humans, associating words to memories and feelings and other human experiences, but he does know that a given symbol evokes the use of other specific symbols. Some sequences require the usage of lots of symbols, some are somewhat ambiguous, and some require him to fetch a symbol that he hasn't used in a long time, maybe doesn't even know where he stored it. To me this looks a lot like the processes that happen inside our minds, with the exception that his form of "understanding" and the experiences that this evokes to him are completely alien to us. Just like an AGI would possibly be.

I'm not confortable looking at the translator's point of view as if he's the analogous to a mind. To me he's the correlator, the process inside our minds that makes these associations. This is not us, it's not under our conscious control, from our perspectives it just happens, and we know today it's a result of our neural networks. We emerge somehow from this process. Similarly, it seems to me that the experience of knowing chinese belongs to the whole room, not the guy handling symbols. It's a weird conclusion, I still don't know what to think of it though...

[siglesias]

When I say "experience," I mean a sufficient grounding of certainty about what a word means, which includes how it's used, how it relates to the world that I'm experiencing, but also the mood or valence the word carries. I can't feel your pain, or maybe you've been to a country that I haven't been to and you're conveying that experience to me. Maybe you've been to outer space. I'm not saying to understand you I need to literally have had the exact experience as you, but I should be able to sufficiently relate to the words you are saying in order to understand what you are saying. If I can't sufficiently relate, I say I don't understand. You can see how this differs from what an AI is doing. The AI is drawing on relationships between symbols, but it doesn't really have a self, or experience, etc etc.

The process of fetching symbols, as you put it, doesn't feel at all like what I do when somebody asks me what it was like to listen to the Beatles for the first time and I form a description.

[mannykannot]

The irony here is that performing like an LLM the very thing that Searle has the human operator do. If it the sort of interaction that does not need intelligence, then no conclusion about the feasibility of AGI can be drawn from contemplating it. Searle’s arguments have been overtaken by technology.

[siglesias]

Can you expand on this? The thought experiment is just about showing that there is more to having a mind than having a program. It’s not an argument about the capabilities of LLMs or AGI. Though it’s worth noting that behavioral criteria continue to lead to people overestimating the capabilities of promise of AI.

[mannykannot]

LLMs are capable of performing the task specified for the Chinese room over a wide range of complex topics and for a considerable length of time. While it is true that their productions are wrong or ill-conceived more often than one would expect from a well-informed human, and sometimes look like the work of a rather stupid one, the burden now rests on Searle's successors to show that every such interaction is purely syntactic.