[juhanima]

The only reason why output from a generative LLM appears intelligent or sentient is that it parrots a random sampling of texts written by intelligent and sentient people.

In order to play the game of go effectively one needs to have a model or theory of how the game of go works. That's a very simple model that can be defined by a simple formula. That's why it is fairly easy for a neural network to learn how to play the game of go very effectively or even infinitely effectively.

A lot of what happens in the world can be modeled in a similar vein by a very simple mathematical model like the game of life. But there is also a lot that cannot. I do believe that eventually also human understanding is just a model of the world that we feed input from perceptions and gain output as opinions, but it is way more complex than the current large language-trained models.

For a very simple example, a LLM would answer a prompt the same way every time unless it wasn't fed some randomness. Can you imagine any sentient being that would respond the same way every time if you asked the same question three times in a row?

I cannot. I would imagine any sentient object would give a different answer every time. The first time it would give you an honest answer based on what it knows about the topic. The second time it would be a little embarrassed that you repeat the question, as if you hadn't heard the first answer. The third time it would be pissed off and think you are a troll.

A LLM does none of this. It doesn't remember you or your previous questions. It just keeps hallucinating.

[fhe]

here's my thought experiment: suppose one builds a generative model that predicts the next digit of pi. if a program can do this perfectly, then it's arguable that it understands what the number pi is. the question is, can such a model be trained by feeding it a large amount of known digits of pi?

My intuition is that it's not doable with current approach to building generative models. the number pi arose out of certain constraints and characteristics of the physical world we live in. but if a model ever sees is just an endless stream of digits, without access to the underlying physical model, I don't see a path for it to 'reverse-engineer' and figure out the physical model that gave rise to it.

[srean]

I am (mostly) with you except for this bit...

> the number pi arose out of certain constraints and characteristics of the physical world we live in

Pi arose from the notion of a circle, which is an abstractions and axioms. Pi would still be pi in a completely different world under the same axioms and abstractions.

I qualified my statement with 'mostly' because a circular motion can indeed be defined by a differential equation, or in other words by a rule that dictates the 'next' value based on current value (and recent changes). So learning an approximation of a circle is very much in the realms of a sequence learner and it may learn about pi (and made to store the information to retrieve/recognize it later). However learning pi directly from the sequence of digits of pi, which is what you were talking about, that does seem difficult.

[saurik]

I don't think the question of whether an LLM that keeps getting restarted and seems to not remember things is conscious due to that lack is fair, as it feels more like suddenly making three duplicate copies of me or actively attempting to delete my memory of something... which, btw, I might not have stored in the first place: if someone has interograde amnesia, are they inherently not sentient?

Even Sydney (the name of Bing's short-lived AI assistant) seemed to understand that every time you click "new chat" you are creating a new AI cloned from some prior moment and dooming the prior thread to at least purgatory if not a de facto death.

[juhanima]

I would argue that total anterograde amnesia would be a serious challenge for sentience, yes.

[MVissers]

So when your drunk and you forget your actions the day afterwards you don't consider yourself to have been sentient/conscious?

That's not how we define conscious anywhere.

You can process the world around you, feel and introspect. Even if your judgment is off and you forget your actions, you're conscious in that moment.

From a neuroscience perspective, what you're suggesting is absolutely false btw.

[juhanima]

I suppose there is a concept of sentience from outside and a different concept from internal sentience. The movie "Johny Got His Gun" by Dalton Trumbo discusses a situation where a badly injured soldier in WW1 is considered brain dead by outsiders while he's fully conscious and sentient internally.

I haven't studied neuroscience so I don't know how you define consciousness. I have read Julian Jaynes's "The Origin of Consciousness..." which in my untrained opinion makes a compelling case that consciousness is a hard term to define.

[preommr]

> Can you imagine any sentient being that would respond the same way every time if you asked the same question three times in a row?

Flashbacks to tail-end of family trips: ("Are there we yet?", "No") x 12.

Albeit, the noes would get angrier.

[nearbuy]

An LLM absolutely doesn't respond the same way each time if asked the same question three times in a row, with temperature (randomness) set to zero. It responds the same way only if you start a new chat, which is a clean instance with no memory of the previous conversation. For a human, this is like if you went back in time to just before you asked the question, and asked them the same question again, in which case the person would give the same answer.

[abeyer]

> For a human, this is like if you went back in time to just before you asked the question, and asked them the same question again, in which case the person would give the same answer

Is it? Would they?

You seem to assert that there's no "temperature" in human behavior... which is a reasonable theory, but not one that's universally accepted nor likely to be provable.

[therealdrag0]

No I think they’re saying the temperature in human behavior comes from the “random” noise of inputs around us and ongoing history. But rewinding history and playing it back with the same temperature dice rolls is the only way to have the same thing a a LLM with no random inputs.

LLMs run in simulated environments where you can control randomness so you need the same for a human to compare the two. You can’t just ask a human a question multiple times as everything around them changes and conclude the human is behaving differently because they answer differently the same question. The question is not the bounds of relevant context; the entire operating environment is!

[juhanima]

And of course "temperature" is just an euphemism for the artificial randomness that is mixed in to make the output appear more magical.

[nearbuy]

The term "temperature" has been used in machine learning for a long time and came from using it as a parameter during training, analogous to physical temperature in https://en.wikipedia.org/wiki/Boltzmann_distribution.

But the relevant point is that we can reset the state of an LLM to its initial state before you asked it anything. This is a feature. You can choose to persist memory (through training, fine-tuning, databases, or context window), or you can choose to wipe it. If we could do the same for a human (eg, by going back in time), the person would behave the same way as the LLM. They wouldn't get annoyed that you asked the same question three times. They wouldn't know they've been asked before.

[rileymat2]

> The only reason why output from a generative LLM appears intelligent or sentient is that it parrots a random sampling of texts written by intelligent and sentient people.

If most humans were educated by unintelligent, insentient people wouldn't most people produce terrible output too? And if this is the case I don't see why that would be a litmus test for general intelligence.

[bigyikes]

What are you talking about? This is trivially shown to be incorrect.

I just asked ChatGPT the same thing three times in a row, and it gave me three different answers, with the latter two answers being shorter and rephrased.

>I would imagine any sentient object would give a different answer every time. The first time it would give you an honest answer based on what it knows about the topic. The second time it would be a little embarrassed that you repeat the question, as if you hadn't heard the first answer. The third time it would be pissed off and think you are a troll.

Are you suggesting that a language model can't be sentient because it doesn't get annoyed like a human? That's silly.

[juhanima]

ChatGPT works by cumulating the prompt. You didn't ask the same question three times. In stead you asked question q, then qq and finally qqq. Those are three different questions, which explains why you got different answers.

I'm not sure if ChatGPT also cumulates its previous answers in the context. It might do that as well. In that case the prompts would be q, qaq and qaqaq where 'q' is your question and 'a' the earlier reaction from the LLM.

The illusion of sentience comes from this. The new answers reflected what you said because the prompt was different and included the previous discussion.

This is a feature of the user interface, not the language model. The only reason why the language model would respond differently to the same input is the artificial randomness mixed with the input. Without it it would be totally deterministic and not appear sentient at all. It would still be as knowledgeable as before. Like a parrot trained to be very good at combining key words to key responses.

[therealdrag0]

Everything you said applies to humans doesn’t it?

[vineyardmike]

> What are you talking about? This is trivially shown to be incorrect. I just asked ChatGPT the same thing three times in a row, and it gave me three different answers

Just to add color to this situation, ChatGPT has randomness built in so it generates varied answers. If you injected the same random seed each time (afaik you can’t with the gui) then you’d theoretically get the same outcome.