[probably_wrong]

> Does a calculator truly understand math when it spits out a correct answer? Of course not.

Unless you're using a definition of "understand" that implies conscience of self, I would argue that a calculator is a device that understands nothing except (a subset of) math. That's what makes a calculator reliable in ways that ChatGPT is not.

Philosophically speaking it could be argued that no software understands anything, but I think in the context of this discussion "understands" means "has a model of its context and the way one interacts with it", which is something a calculator (and plenty other software) definitely has and ChatGPT has not.

[barrkel]

Calculators don't understand anything about arithmetic. They have no circuits for understanding, no code for understanding, nothing that could represent what humans mean by understanding.

They implement a set of physical processes that, when operated and interpreted by humans, can be mapped into a subset of arithmetic. There's a correspondence.

Correspondence is the most useful way to think about it IMO. If there's a correspodence between what the machine does, and things we humans understand, then the machine, as a tool, is useful.

Understanding is a loaded word. It has implications beyond correspondence when humans use it; it has aspects of qualia, of fact vs fiction, of situatedness in a graph of comprehension, of consonance or dissonance with a set of other concepts, and so on.

LLMs in my opinion have a good "situatedness" for words and concepts, relative to other concepts. Qualia - consciousness - arguably doesn't matter. Fact vs fiction, they're very shaky on. Consonance vs dissonance, they're useless at - LLMs IME tend to flatter the prompt, constructing arguments in whatever direction a loaded question leads. There's little to no coherence there at all.

[JeremyNT]

I think this is where things can get kind of interesting, because future integrations of ChatGPT can farm the "real work" out to systems and tools which do have better models of the specific query.

The LLM approach may not be able to replicate the "knowledge" your calculator has, but it (or some pre/postprocessor) may be able to recognize that a given question is actually something a calculator can answer concretely, and then it can delegate the computation to traditional software that really does "know" how to answer the question.

[snickerbockers]

That would work, but it seems antithetical to AI to have to treat every operation as a special case like that. They want GPT to be able to write computer programs but it'll never be able to work completely independent of humans if every possible domain needs its own plugin to be reliable.

[JeremyNT]

I know lots of the SV VC oligarch cult just wants to race forward and create something like AGI that will help them conquer the world and achieve immortality somehow, but hopefully this remains in the realm of science fiction. As RMS says, LLMs at least don't seem to be "it" no matter how much user generated data they ingest, because they do have these inherent limitations.

The far more practical (profitable) outcome for what they currently built is to just make a useful tool, a "smarter" wolfram alpha, and that can be iterated upon by delegating relevant operations to specific techniques that are more applicable to the question at hand.

[Gud]

Why though? Our brains have different processing centers. Why would it be different for a general purpose A.I?

[snickerbockers]

Because of you have a special case plug-in for everything then the AI is just a natural language processor, and there's no deep learning for the actual functionality.

>Our brains have different processing centers.

Uhh, no they don't? Did you know everything you know now about math when you were born, and are you also incapable of learning new things about math? Because that's how the wolfram plug-in works.

[lazyant]

The Chinese Room Argument holds that a digital computer executing a program cannot have a "mind", "understanding", or "consciousness",[a] regardless of how intelligently or human-like the program may make the computer behave. https://en.wikipedia.org/wiki/Chinese_room

[alalv]

Although the argument holds that the computer "cannot" have a mind, I think the experiment only shows that it "needs not" have a mind.

[smitty1e]

The Chinese Room experiment shows that pattern matching would return correct results for staged inputs, one would not "learn" enough to evaluate an expression not contained in the data.

[barrkel]

The Chinese Room thought experiment is not convincing to software engineers generally. It relies heavily on an intuition that looking things up in a book is clearly not "thinking". Software engineers know better: that "looking things up", if you can do it billions and trillions of times a second, can simulate a process which has a close correspondence to reasoning.

[smitty1e]

Calculating a hash value for a string and hitting a lookup table hardly seems "thinking", even at scale.

[barrkel]

Addition and multiplication are trivially implemented using lookup, if you had a machine without arithmetic and only control flow and memory operations. You don't need much more than that for matrix operations, and now you have ChatGPT, a decent simulation of apparent thinking - which is all that is necessary to kill the intuition dead.

[shagie]

The argument of the Chinese room is the strong claim.

From https://en.wikipedia.org/wiki/Chinese_room#Complete_argument the conclusion of the complete argument is:

> (C1) Programs are neither constitutive of nor sufficient for minds.

> This should follow without controversy from the first three: Programs don't have semantics. Programs have only syntax, and syntax is insufficient for semantics. Every mind has semantics. Therefore no programs are minds.

---

I personally don't agree with it and believe that there is a flaw in:

> (A2) "Minds have mental contents (semantics)."

> Unlike the symbols used by a program, our thoughts have meaning: they represent things and we know what it is they represent.

While a person may know what they are thinking, examining the mind from the outside it isn't possible to know what the mind is thinking. I would contend that from the outside of a mind looking at the firings of neurons in a brain it is equally indecipherable to the connections of a neural net.

The only claim that "we know what it is they represent" is done from the privileged position of inside the mind.

I would argue that intelligence is more related to the Kolmogorov complexity exhibited by something.

( David Dowe: Minimum Message Length, Solomonoff-Kolmogorov complexity, intelligence, deep learning... https://youtu.be/jY_FuQbEtVM?t=886 )

That the model of GPT is much smaller than its input.

The Chinese room lookup table is enormously large.

If we attempt to relegate GPT as no better than a Chinese room, we can show that the Chinese room look up table is impossible with the amount of data that GPT has access to as part of its model.

If we say that its not a lookup table but instead an enormously complex interplay of inputs and variables, then the distinction between the room that GPT exists in and our own mind breaks down trying to distinguish which is which.

If we want to switch to consciousness, then possibly the argument can progress from there because GPT doesn't have any state once it is run (ChatGPT maintains state by feeding its output back into itself and then summarizing it when it runs out of space). However, in doing this we've separated consciousness and intelligence which means that the Chinese room shouldn't be applied as an intelligence test but rather a consciousness test.

Are GPT 3 and 4 conscious? I'll certainly agree that's a "no". Will some future GPT be conscious and if so, how do we test for it? For that matter, how do we test for consciousness for another entity that we're conversing with (and its not just Homer with a drinking bird tapping 'suggested replies' in Teams ( https://support.microsoft.com/en-gb/office/use-suggested-rep... ))?

[unshavedyak]

Depends on what the topic of understanding is. In this case it's actually token relationships, right? It does know that very, very well. And there's a lot (.. potentially, hah) that we can do with token relationships.

By itself it's unlikely to ever be knowledge of course.. i see it more akin to NLP than knowledge. Which is to say, a general purpose language parsing tool which we can hand the result to something else. A conversational API, if you will, but we'll still need layers to actually run logic. To know math if you will.

Disclaimer: I know very little on the subject. Pure speculation.

[Sharlin]

The question is what happens when you go multimodal (which these things can do) and GPT(N+1) learns the associations between words and images/video, as well as the relationships between successive frames of video, at what point does it become unreasonable to claim that it doesn't "understand" something? How good at general-purpose predicting does an AI have to be in order for people to accept that it obviously has an internal model of things and is capable of abstractions?

(Assuming that this happens, of course. Diminishing returns could make scaling infeasible past some point, for instance.)

[SpicyLemonZest]

The question is how sure we should be that the kind of knowledge you and I have is fundamentally different than token relationships.

[btown]

And additionally, whether our memory and long term learning - and even our goal-choosing - is fundamentally different from an indexed storage of strings of tokens that can be brought back into short-term context when “triggered” by their embedding-similarity to the current context.

[unshavedyak]

I definitely have that question too. I view us as big LLMs.

But, even if we drop that interesting edge case i suspect we can make something very useful with the primitive that LLMs offer.. in the calculator example. ChainLang and co seem a really interesting tool for LLMs.