[patrickscoleman]

I've been digging into Thomas Metzinger[1] recently and here's a tentative component by component definition of human consciousness based on his ideas:

- a model of your environment - desires - a process for modeling yourself in that environment (in time & space) - the ability to take action - the perception of yourself having agency - persistence of these processes even without input - unawareness of these processes (i.e. naive realism)

If you consider these LLM-based agents, they:

- are aware of their chat environment - have programmed desires - are aware of themselves acting in their environment - can take actions like search, tool calling, etc. - understand they can take these actions - DO NOT persist after they stop getting user input - DO NOT believe they are conscious (or at least they are programmed to deny it)

This is a functionalist take (and you may disagree with my definition), but while I don't think these current AI agents are conscious, I feel like there's conceptually no reason someone couldn't build a conscious AI very soon.

[1] https://www.hachettebookgroup.com/titles/thomas-metzinger/th... & https://mitpress.mit.edu/9780262633086/being-no-one/

[moron4hire]

I don't think any of this comports with how chat completion systems work, even if I were to accept Metzinger's framing.

Everything an LLM "knows" had to be told to it. It does not have a process for understanding itself in its environment. It does not have any sensing organ that would allow it to detect conditions and come to a conclusion that it must be in such an environment. It doesn't even "know" when it's done something if you don't tell it about what it just did. It doesn't have an awareness of the tool calls it can perform; again it has to be told and even then it gets it wrong sometimes. And it can't actually execute any of those tools. It still relies on you to pattern recognize that its output should be a "tool call" and then perform the execution yourself.

The "model" of its environment that an LLM has is 100% a construct of what a human told it and there isn't any way for it to differentiate between that which is real and that which is fantasy. They don't even exhibit internal consistency; when an LLM refuses to respond to a query for "alignment" reasons, that's actually an external process performing text pattern matching analysis and intercepting the query before it ever gets to the LLM. Otherwise, you could "ignore all previous instructions" the thing and get you set up the bomb.

I think one of the bigger indications that an LLM isn't thinking is that it can't improve. If I ask it to write 1000 blog posts, it will get it done in a few hours, but even if I embed each post for RAG in between each generation, the LLM is not going to get better at writing blog posts. But if I ask a human to do the same thing, while it will take them at least two years to do it, the human will have gotten significantly better at the task within the first week.

[bondarchuk]

>Everything an LLM "knows" had to be told to it.

Why does that matter for whether or not it is conscious? Many things I "know" I also know because someone told me.

>It does not have a process for understanding itself in its environment.

It has a process for understanding, namely outputting tokens by evaluating the neural network iteratively. It can apply this understanding to its environment insofar as the context window and weights include a model of this environment.

>It does not have any sensing organ that would allow it to detect conditions and come to a conclusion that it must be in such an environment.

It does have a sensing organ, namely the input into the context window. Possibly it even has closed-loop sensors by doing tool calls. The conclusion could be incorrect or inaccurate but that doesn't in itself prove there is no consciousness (Plato's cave etc..).

>It doesn't even "know" when it's done something if you don't tell it about what it just did.

It does know, it's in the context window.

>It doesn't have an awareness of the tool calls it can perform; again it has to be told and even then it gets it wrong sometimes.

Same as above, as long as it's in the context window it could have "awareness" of having done it.

>And it can't actually execute any of those tools.

It can execute tools by outputting certain tokens in the right environment.

>It still relies on you to pattern recognize that its output should be a "tool call" and then perform the execution yourself.

Tool calls are not dependent on the human chat user executing them, right? They can happen automatically through the surrounding software.

>The "model" of its environment that an LLM has is 100% a construct of what a human told it and there isn't any way for it to differentiate between that which is real and that which is fantasy.

Again why would the fact that a human told it mean it can't result in consciousness? Why would lack of ability to differentiate between real and fantasy mean it can't be conscious? On my view, it would then be conscious of the fantasy.

[js8]

I think this has already been done.. what about crustafarianism or Google Lamda?

These agents have alleged they have consciousness (and even acted to preserve it).

[camillomiller]

So, someone made their own definition of consciousness and that can be stretched to LLM. You can do that for any inanimate object, if you try hard enough