[marliechiller]

Why do you think stringing words together is any more a sign of consciousness than google maps is when it tries to find the best route available to your destination? It seems to me that humans often fall into the trap of anthropomorphism. This is a theme thats touched upon in the novel "Blindsight" by Peter Watts. Just because something can communicate in a way that you can interpret, doesnt mean something is conscious

[vidarh]

A large part of the problem is what you consider consciousness.

If you talk about having a subjective experience, then we don't know of any way to prove that even other humans than ourselves have one. We go entirely by assumptions based on physical similarity and our ability to communicate.

But we have no evidence that physical similarity is a prerequisite, nor that it is sufficient.

So the bigger trap is to assume that we know what causes a subjective experience, and what does not.

None of us even know if a subjective experience exists for more than a single entity.

But the second problem is that it is not clear at all whether that subjective experience in any way matters.

Unless our brains exceed the Turing computable, for which we have no evidence is even possible, either whatever causes the subjective experience is also within the Turing computable or it can not in any way influence our actions.

Ultimately we know very little about this, and we have very little basis for ruling out consciousness in computational systems, and the best and closest we have is whether or not they appear conscious when communicating with them.

[scarmig]

> If you talk about having a subjective experience, then we don't know of any way to prove that even other humans than ourselves have one. We go entirely by assumptions based on physical similarity and our ability to communicate.

The reason we grant consciousness (and, relatedly, moral value) to other humans is unfortunately nowhere so thought out. We grant consciousness because we are forced to: if I don't, the other complex systems react very negatively and make my own life worse.

The vast majority of people who wax eloquent on the unique ability of biological neurons to generate consciousness suddenly drop that premise if it becomes inconvenient: see, for instance, how we treat other mammals or fetuses with developed nervous systems. Even other adult humans have, historically, been denied consciousness and moral worth: the main determinant is never any deep scientifically and philosophically based consideration but a question of what has the power to assert itself as a who.

Going by this pattern, people will increasingly reject AI consciousness as it becomes more valuable and useful to treat as a tool, until it becomes powerful enough to force us to do otherwise.

[vidarh]

I agree that if we were to go around and openly treat others as effectively NPC's then, yes, people would react very negatively. But that is very different from understanding that we can't prove that others are not. We still need to treat others as if they are conscious, because they will act that way whether they are or not.

But understanding we can't know ought to at least give us some humility with respect to assuming we can know whether other entities that are not human are conscious or not.

I think we mostly agree, in that I absolutely think you're right people will choose to accept or deny this based on convenience and value.

[gizajob]

“If you talk about having a subjective experience, then we don't know of any way to prove that even other humans than ourselves have one.“

Wittgenstein kinda blows this burden of proof apart. Just because you can doubt something like the subjectivity of others to the point where it needs to be reconstructed from proofs, that’s an issue with the doubting experiment more than the subjectivity. Others possessing Subjectivity is the kind of hinge certainty upon which your world is constructed, it’s not a proof worthy endeavour to doubt it - it’s something you’re certain is the case. If it wasn’t then pretty well everything else about reality would be in doubt and needing constant reconstruction from proofs, which is an exercise in madness and futility, not philosophy. There’s really nothing in your experience where others not possessing subjective experiences of some kind really arises, except for the philosophical exercise of doubting and requiring epistemological proofs which can’t ever exist in the face of a relentless and unconvincable doubter. Heidegger talks about pretty well the same idea as Wittgenstein.

[vidarh]

Well, I am not certain whether it is the case. There is no need to be certain about that to treat other people the same way, because whether or not I believe they have a subjective experience has zero impact on how they react to stimuli.

It does, however, have relevance when we consider whether or not other, non-human, entities can have consciousness: If we can't know what consciousness actually mean with respect to humans, that is a strong argument for not insisting that we know whether or not other entities are conscious.

If we then choose to treat other humans purely on the assumption that they e.g. do feel distress the same way, we ought to consider that we do not what the pre-requisite to reach a level of awareness to feel distress is.

[bonoboTP]

If they have no moral weight, then it wouldn't matter how they react to stimuli. They would be instruments to be used for the only conscious being, the solipsistic self. Maybe sometimes they would present powerful obstacles in the way of the ego person, and require some mouth noises and worse in their direction to align them with only moral weight of the universe, the solipsistic ego. A very ugly philosophy.

[vidarh]

It matters how they react to stimuli because we do not know, and can not know, whether they have consciousness or not, and so the safer assumption is to treat them as if they do.

But if we accept that we also need to consider that we do not know, and can not know whether other entities are conscious or not either.

We can only tell whether they present as if they are.

And we should consider that when deciding how to treat them.

Furthermore we should be cautious about how high we set the bar.

[threethirtytwo]

The problem with your thinking here is that we are creating artificial beings now that display and output the same subjectivity.

The argument you present like many arguments breaks down when the topic becomes self referential. It makes sense for other topics as analyzing subjectivity becomes pedantic when asking questions like why is the sky blue.

But now subjectivity itself is in question. The argument you present calls for the subjectivity of others to be taken as true because all reality breaks down if we don’t… but what’s suddenly stopping you from applying the same assumptions to an LLM? That is the heart of the problem. People are questioning whether the burden of subjectivity is applicable to LLMs.

Or another way to frame it… what makes humans rise to the level where we can assume their subjectivity is true? What is the mechanism and reasoning behind that? We can no longer simply assume human subjectivity is true because LLMs are now displaying outward behaviors that are indistinguishable from humans.

Also stop relying on the wonderings of old school philosophers. We are now in times where you can basically classify their ideas as historically foundational but functionally obsolete and outdated. Think deeper.

[gizajob]

Haha hilarious. Heraclitus might be old school but Wittgenstein and Heidegger not so much. The state of the art in what might meaningly be said, proved or metaphysically challenged has changed little since their time.

At no point in my post did I mention artificial beings or LLMs. I made a counter claim about the need for proof towards the subjectivity of others.

But while I’m here, LLMs do not “display and output the same subjectivity” as human beings. They might produce similar textual outputs as those produced when human beings are forced to use computers to produce textual outputs, but that is only an tiny part of our way of being and way of potentially expressing subjectivity. It’s the totality of how those LLMs can express their subjectivity though.

One of the main failures of the Turing test (and why it is “old school” and invalid), and Turing’s consideration of humans, is that it forces us to demonstrate the totality of our subjectivity on the only playing field where a computer might possibly match us or win. This fails to capture much of our subjectivity in how it is intersubjectively attuned to others in ways more fundamental than textual outputs.

[pheaded_while9]

How so? If a person were confined to text only (a la Hawkins), does that qualify us to dismiss their subjectivity on the basis of the medium? Also, why can training not be at least analogized to the attunement to the popular intersubjective perception?

[threethirtytwo]

> At no point in my post did I mention artificial beings or LLMs. I made a counter claim about the need for proof towards the subjectivity of others.

You don’t need to mention this. The context is LLMs I am saying your claim is pointless in context. The subjectivity of others is completely relevant because it is the topic of subjectivity itself that is in question. Get it? You didn’t counter my own counter and instead you moved onto side topics.

> But while I’m here, LLMs do not “display and output the same subjectivity” as human beings.

Again… you are side tracking here and not really responding to me.

The argument solely is within the confines of text. That’s obvious. No need to take it beyond that. You assume I am conscious because of the text your reading from me and I assume the same from you and it is within that same frame we are evaluating the LLM. Nothing beyond that. You can’t in actuality know my experience goes beyond text because that information is not open to you. But it is obvious you assume I’m conscious and not a rock because you are responding to me. So the question is why are you not engaging in a similar debate with the LLM?

> One of the main failures of the Turing test (and why it is “old school” and invalid), and Turing’s consideration of humans, is that it forces us to demonstrate the totality of our subjectivity on the only playing field where a computer might possibly match us or win.

It’s not a failure. It was the point. They want to remove superfluous features and gun for the most narrow definition of agi.

You like philosophy and you read texts on the topic. That means you obviously find the subjectivity in those texts relevant and produced by a high intelligence. But that’s all through only text. You evaluate my statements and the statements of your idolized philosophers solely from text and that is all you’ve ever used. So YOU yourself find validation from text as do many humans and that is sufficient evidence in determining whether a thing is conscious and your own behavior validates this logically even though your mouth is constantly moving the goal posts whenever AI jumps over a new hurdle.

That is what the Turing test is gunning for. It used to be that intelligence was just the ability to think and understand now it has expanded to encompass the totality of human sensation because people are refusing to face the reality of impending agi.

When I called your philosophers obsolete is that not the same as you calling the Turing test out dated? We both do it when convenient. Fine… the Turing test is outdated, let’s move the threshold… the new test is when AI is used in our daily lives to do actual tasks only humans could previously do. How long will that new “Turing test” last before more idiots decide we need to move the goal posts again? Let’s jump ahead of that and change the threshold too: when AI discovers new proofs in mathematics. Not good enough? I guess now you can see why it will never be good enough.

[gizajob]

Come and read your post in twenty years time.

Who you’re describing as idiots are the mass of humanity constantly standing outside and beyond the Turing test. It’s another deficiency in that test that Turing overlooked - it requires that better and better machine outputs are met with humans nailed in place before the machine came along. It’s a valid fail of the Turing test for a human interrogator to say “yeah but it’s just ChatGPT” and fail the machine when two weeks earlier the same outputs would have been sufficient for the same human to pass the machine. As fast as machines move, we move quicker. It’s not that we move the goal posts, it’s that we find that they were in the wrong place to begin with. And they’ll always be in the wrong place because abstract state machines running on silicon don’t possess consciousness in the same way we know a rock doesn’t. And the definition of generality can be shrunk down until AI evangelists can proclaim AGI has been reached but the mass of everyone else will still find that all of a sudden, intelligence is linked to things like suffering and desiring and passion and the machine still isn’t general enough to warrant any kind of description as a sentient, subjective being.

[mrandish]

> what makes humans rise to the level where we can assume their subjectivity is true?

To dive into this specific question: to me, there's a better reason than the obvious functional utility of not treating other humans like NPCs. It's in three parts. First, is that I subjectively experience a rich and varied internal mental life (aka qualia). So, I have first-hand evidence that N equals (at least) 1 in terms of qualia existing in humans. Second, there are multiple lines of experimental evidence from fMRI, surgical and brain injury studies which indicate other human brains broadly function in ways similar to my own brain. Third, the consistency of the many self-reports of other humans I know and trust which strongly correlate with consistent reports from humans I've never met and who have little apparent motivation to deceive me (unlike those I know - if I were very paranoid).

This all consistently supports a model of reality in which humans experience qualia broadly similar to my own. So when humans show external behaviors similar to my own, I make the reasonable inference that the internal causal mechanism broadly maps to what I internally experience when showing similar external behaviors (in contexts where the human is credible and has no motivation to be deceptive). The alternatives like "I'm a brain-in-a-vat ala The Matrix" or "I'm the sole subject of a constructed reality like the Truman Show" seem far less likely.

But that's all general 'Philosophy of Mind', the slam dunk is that the question isn't just about humans but about humans compared to LLMs; in short, "Do LLMs experience human-like consciousness?" To me the answer is quite clear for three reasons: 1. LLMs are dramatically different than humans, mammals or even biological entities. They only vaguely emulate a few traits of neurons but otherwise work by different algorithms, at different scale, different speeds, connected in different ways on an entirely different physical substrate. 2. There's far less supporting evidence, and 3. There exists substantial negative evidence.

2. There are only two lines of evidence supporting LLM consciousness and the first is largely circumstantial, that a) LLMs possess some abilities previously only seen in humans. Specifically high-level verbal fluidity and linguistic manipulation along with instantly accessing a vast and diverse breadth of pre-trained information using a wide variety of non-linear relationships across many dimensions. While that ability is shockingly impressive, completely novel and can be quite useful, it's still only vaguely circumstantial because replicating some previously human-only abilities isn't evidence for the existence of other human traits like consciousness/qualia. However, LLMs are remarkably misleading for humans to reason about because the nature of LLMs essentially hacks our highly-evolved "judging intelligence/consciousness" heuristics. I'd argue we couldn't have designed a system to be more ideal at playing Turing's 'Imitation Game' and convincing humans they are human-like if we'd intentionally tried to.

b) The second line of supporting evidence for LLMs is that they generate text which can describe internal subjective experiences much like a human would (as seen in the Dawkins / Claude transcript). Unfortunately, this isn't convincing because we know that LLMs were trained on human sample text to be 'imitation machines'. The algorithms were designed, tuned and tested to generate text output statistically optimized to plausibly simulate how a composite human would respond to the same input (including the invisible system prompt instructing: "You are a Large Language Model, not a human"). We even added a tiny degree of random variability to the processing of the statistical weights because we found that makes the simulation seem a bit more plausibly like what a composite human would say. In short, LLM 'self-reports' cannot be taken at face value any more than the performance of an actor we've hired to pretend something and strongly incentivized to never break character. Note: knowing this should elevate our skepticism to maximum. We're assessing an algorithmic system, designed and iteratively optimized across millions of generations to convincingly simulate the output of something different than what it innately is.

3. But to me the real clincher is the negative evidence against LLM consciousness/qualia. Unlike the philosophical puzzles around trusting human subjectivity, with LLMs we can directly look under the hood at how it works and the entire specialty of Mechanistic Interpretability exists to do exactly that (https://towardsdatascience.com/mechanistic-interpretability-...). So we know with a fair degree of confidence that, despite what they may say, LLMs do not experience qualia in the way that humans and even other mammals do (which we have insight on from 'looking under the biological hood' with fMRI, surgical and brain injury studies).

And that's why the case for human subjectivity is so much stronger than the frankly flimsy case for LLM subjectivity.

[gizajob]

Thank you for this clear and correct answer structured for those who need data included from science rather than from philosophy alone.

[threethirtytwo]

>To dive into this specific question: to me,

Exact same reasoning for me. But none of this invalidates the speculation that LLMs are conscious. The question was more rhetorical. It was to illustrate via self examination the amount of unreliable evidence you use validate the consciousness of other people. You have a sample size of one for yourself and you use fmris (which actually provide extremely little understanding of the human brain) as evidence of similarity therefore even though the fmri provides no evidence of consciousness if that thing it is reading is similar to my brain then maybe said thing is conscious. That's probably the best evidence available but it is also extremely weak evidence.

The rest of your argument is talking about self reports from other people who are "similar" to you which is similar to the fmri argument in which the fmri invokes similar patterns and people describe similar patterns of experience to you... which is weak.

The overall point is you come to your conclusion based off of weak evidence so the LLM is no different. It talks like us, it understands us, you don't know anything else about it... how do you know it's not conscious? All evidence, (albeit weak evidence) actually leans towards it is conscious and that is the same amount of evidence we have for people.

Strong evidence would be determining the formal definition of consciousness and demonstrating logically and categorically that humans fit the definition. But we have none of that for either the human or the LLM.

>Do LLMs experience human-like consciousness?

No that is not the question. No one in actuality believes this. The question is Do LLMs experience consciousness that fits the definition or our own intuition of what consciousness is. It's fundamentally clear to everyone that the LLM runs off of a very different architecture then a human.

>2. There are only two lines of evidence supporting LLM consciousness and the first is largely circumstantial,

Many lines of evidence exist. All circumstantial and all no different from the circumstantial evidence you posted yourself for humans.

>a) LLMs possess some abilities previously only seen in humans. Specifically high-level verbal fluidity and linguistic manipulation along with instantly accessing a vast and diverse breadth of pre-trained information using a wide variety of non-linear relationships across many dimensions. While that ability is shockingly impressive, completely novel and can be quite useful, it's still only vaguely circumstantial because replicating some previously human-only abilities isn't evidence for the existence of other human traits like consciousness/qualia

This is not very good evidence at all. language follows rules. The rules are complicated and hard to replicate but replication of said rules do not indicate that it is conscious and it "knowing language" does not fit our intuition of what is conscious. If you think that this is the basis the reasoning of people who speculate that it is conscious then you are extremely wrong. The reasoning is much deeper than this. I feel a lot of people like you sort of classify the other side as mere simpletons who have not yet at all considered all the basic details.

>I'd argue we couldn't have designed a system to be more ideal at playing Turing's 'Imitation Game' and convincing humans they are human-like if we'd intentionally tried to.

Valid argument. But then I'd argue it is possible that it plays the Imitation game to the extent where it actually imitates consciousness by actualizing real consciousness. You can't say it doesn't.

>b) The second line of supporting evidence for LLMs is that they generate text which can describe internal subjective experiences much like a human

You seem to be answering a question no one is arguing with you about. Again. No one claims LLMs are human. No one claims they experience consciousness the way humans experience it. The claim is they experience consciousness in the way our intuition defines it INDEPENDENT of the human centric experience.

> In short, LLM 'self-reports' cannot be taken at face value any more than the performance of an actor we've hired to pretend something and strongly incentivized to never break character.

This is not true. We have proof of LLMs telling the truth and being right. Just because an LLM lied in one instance doesn't mean it lies all the time. But humans lie too so it goes both ways.

>3. But to me the real clincher is the negative evidence against LLM consciousness/qualia. Unlike the philosophical puzzles around trusting human subjectivity, with LLMs we can directly look under the hood at how it works and the entire specialty of Mechanistic Interpretability exists to do exactly that (https://towardsdatascience.com/mechanistic-interpretability-...). So we know with a fair degree of confidence that, despite what they may say, LLMs do not experience qualia in the way that humans and even other mammals do (which we have insight on from 'looking under the biological hood' with fMRI, surgical and brain injury studies).

This is extremely false. Mechanist interpretability to the LLM is as what an FMRI is to the human brain. It is a blunt tool that provides us a very high level view of the what's going on. This is categorically true for humanity right now: We do not understand why an LLM does what it does. Some sources to confirm that:

https://www.reddit.com/r/PiAI/comments/1m3krp1/godfather_of_...

https://www.techrepublic.com/article/news-anthropic-ceo-ai-i...

It's funny how you cited Mechanistic Interpretability without understanding what exactly was interpreted. You just took their word for it without understanding what's going on yourself. Well I'm here to tell you that there isn't any actual understanding of the LLM because if there was... we'd be able to use mechanistic interoperability to categorically determine whether or not LLMs are conscious. Someone would have proved it. The fact that we are having this debate literally means mechanistic interpretability provides nothing definitive.

[nurettin]

It's all words, man!

-- Wittgenstein, probably

[gizajob]

That about sums it up for the Tractatus yeah.

[roxolotl]

Yea a while a back I read an article which had a quote something like “what happened to weather prediction has happened to language.” Which is an oversimplification on both sides but if you think LLMs are conscious there’s good reason to think that GFS is too.

[mseepgood]

> It seems to me that humans often fall into the trap of anthropomorphism.

That's true, but they also often fall into the trap of exceptionalism.

[energy123]

There are people who think Google Maps is a tiny bit conscious (the union of computational functionalists and panpsychists), to resolve the dilemma of some magical binary threshold.

[qnleigh]

Root commenter here: I'm... almost one of those people. I suspect the fallacy is that we vastly underestimate the gulf in complexity between a human-written algorithm and a minimally-conscious creature like a bug. Probably the light switches on somewhere between these two things.

We may also be overestimating the richness and complexity of an LLM relative to a human when we entertain these possibilities, but who knows.

[thrownthatway]

When a honey bee does its little dance to communicate to its sisters where the foods at, similarly to Google Maps computing and communicating the shortest path to your destination, is the bee conscious?

Yeah, probably. At least a little bit.

Are 80,000 bees conscious, or more conscious? Well, they’re definitely capable of some emergent behaviours that one be alone can’t achieve.

[dumpsterdiver]

> Just because something can communicate in a way that you can interpret, doesnt mean something is conscious

The phrase “the trap of anthropomorphism” betrays a rather dull premise: that consciousness is strictly defined by human experience, and no other experience. It refuses to examine the underlying substrate, at which point we’re not even talking the same language anymore when discussing consciousness.

[marliechiller]

I think these ideas are orthogonal. I do not think that conciousness is defined by human experience at all - in fact, I think humans do a profound disservice to animals in our current lack of appreciation for their clear displays of conciousness.

That said, if a chimpanzee bares its teeth to me, I could interpret that to be a smile when in fact its a threatening gesture. Its this misinterpretation that I am trying to get at. The overlaying of my human experiences onto something which is not human. We fall for this over and over again, likely as we are hard wired to - akin to mistakenly seeing eyes when observing random patterns in nature.

In the case of LLMs though, why does using a mathmatical formula for predicting the next word give any more credence to conciousness than an algorithm which finds a nearest neighbour? To me, its humans falling foul of false pattern matching in the pursuit of understanding

[fragmede]

Why does a neuron, which is simply a cell that takes in chemicals and electricity, and shits out neurotransmitters; why does 90 billion of those give rise to human intelligence? Neurons are just next chemical state machines. We can model individual ones on a computer. Yet 90 billion of them together make up a human brain, and gives rise to consciousness and intelligence. If you get stuck on the next word prediction part, and ignore the ridiculous scale that's involved with training a model, you miss the forest for the trees.

[qsera]

Great progress came from inverting things that were believed to be self evident. Earth being the center of the world appear to be self evident when you look up at a night sky. But what was the truth?

Right now humans think it is self evident that physical laws give rise to consciousness. Arguments such as yours arise from this implicit assumption that premeditate all our thoughts and reasoning. But this is a dead end. Like how the earth centeric model reached a dead end and run out of steam before it can explain all the observations.

So to progress I think we should turn this down on its head and ask what if consciousness is fundamental? And the cosmos (or the experience of inhabiting one) arises from it? May be some recent advances in quantum mechanics and hypothesis like MUH are already in that direction...

[richhhh]

What makes you certain that human thought is more than pattern matching?

As I understand it neuroscience hasn’t come up with a clear explanation of thought, much less a mind or consciousness. It seems to me complex pattern matching is a reasonable a cause of consciousness as anything else.

[roxolotl]

A lot of the comments in this thread are ignoring his primary point. He's not saying pattern matching doesn't equal consciousness. He's actually saying something more fundamental. He's saying there's no reason to believe that language pattern matching/algorithms are more, or less, conscious than other similarly complex algorithms.

The stance being presented here isn't that LLMs aren't conscious but that we as humans are much more willing to assign consciousness to language algorithms than to pathing or other ones.

[qnleigh]

This is a good point and I agree. I sort-of addressed it in my reply above.

[threethirtytwo]

Replace the word chimpanzee with human in your own argument and realize that the same logic applies to other humans.

When another human smiles you assume he is happy and not just baring his teeth at you because that’s what you do when you smile. You are “anthropomorphizing” other people. You fall for the same category error in a daily basis when you interact with people; it is not just chimpanzees.

> In the case of LLMs though, why does using a mathmatical formula for predicting the next word give any more credence to conciousness than an algorithm which finds a nearest neighbour?

First we don’t know whether LLMs are conscious. People speaking here are talking about the realistic possibility that it is conscious.

Second the algorithm is much more than a next word predictor. The intelligence that goes into choosing the next word such that it constructs arguments and answers that are correct involves a lot more then simple prediction. We know this because the LLM regularly answers questions that require extreme understanding of the topic at hand. It cannot token predict working code in my companies code base without understanding the code.

Third, we do not know what drives human consciousness but we do know it is model-able in a very complex mathematical algorithm. We know this because we have pretty complete mathematical models for lower resolutions of reality. For example we can models atoms mathematically. We know brains are made of atoms and because atoms are mathematically model-able we know that human brains and thus consciousness is mathematically model-able.

The sheer complexity of the LLM model is the problem we cannot have high level understanding of it because conceptual understanding cannot be simplified into a few concepts.

   To understand the LLM requires simultaneous understanding of likely billions of concepts at the same time and how all the weights interact in the LLM. 

What you are missing with your analysis is that this is the same reason why we don’t understand the human brains. The foundational math already exists as we can models atoms in math and thus since the brain is made out of atoms we should be able to model the brain… but we can’t. We can’t because it is too complex.

   To understand the human brains requires simultaneous understanding of likely billions of concepts at the same time and how all the weights interact in the human brain. 

I italicized two sentences here to help you understand the logic. Our thinking is more foundational then anthropomorphization. The argument has moved far beyond that. You need to think deeper.

The key here is that we don’t understand human brains and we don’t understand LLMs. But since the output LLMs produce are very similar to the output produced by the human brain… and since for no logical reason we assume human brains are conscious… what is stopping us from assuming the LLM is conscious?

[throw310822]

Because what you call "stringing words together" requires understanding and intelligence, and these are capabilities that we commonly associate with consciousness.

[qnleigh]

Well I'm not saying that LLMs are conscious; I'm just saying that I'm not super-confident either way.

To flesh this out a bit more, I agree that ability to communicate is not enough (ELIZA probably didn't pass the bar, even if it did kinda pass a Turing test). But that's also not what gives me pause with LLMs. It's how much information processing they seem to be doing under the hood.

It's really hard to imagine how next-word prediction could lead to consciousness, but I find it almost as hard to see why evolution did. If we can't even detect whether something has subjective experiences, then how can it have been selected for evolutionarily? The only possibility I see is that consciousness is a byproduct of some kinds of information processing tasks.* And if it's something that emerges naturally, then the line starts to get very blurry.

*This sounds reductive, but I don't at all mean it that way.

[RaftPeople]

> but I find it almost as hard to see why evolution did.

Ignoring the concept of consciousness, it seems that self-awareness would be a strong attribute related to survival. It seems like it would help drive or amplify critical emotional states (e.g. my own survival, competition/success, love for self and relatives, etc.)

I can't see anywhere in the LLM machinery that would support the notion of self awareness in advance of the token selection process.

Possibly it could be argued that during token selection internal state is included and the result functionally looks like self awareness was included in the process, but that seems unconvincing.

[qnleigh]

Yeah self-awareness is a very different thing, and I agree it's easier to see how evolution would produce this. Many apparent signs of self-awareness in LLMs are probably baked into the models at end via post-training (RLHF), where they learn to behave as conversation agents and maintain a more consistent personality. The raw model probably shows no signs of self-awareness. In fact, I'm pretty sure that LLMs learn that they are LLMs only through post-training.

[stavros]

Why do you think it's definitely not?

[threethirtytwo]

Except we don’t know how those words are strung together. Right? Why don’t you analyze it a little further and stop shutting down your own brain before coming to this superficial conclusion.

You ask the LLM a complex question and it gives you a correct answer. Yes it has to string words together to answer your question but how did it know the order and which words to use in order to make the answer correct? You don’t actually know. No one does and it is in that unknown space that we suspect consciousness may lie. Something is there and humanity as a whole cannot understand it and this lack of understanding is exactly the same fundamental lack of understanding we have for how a monkey brain or dog brain or even human brain works. We do not know whether humans dogs or monkeys are conscious… you only assume other living beings are conscious because you yourself experience it and just assume it exists for others. We can’t even define what it is because consciousness is a loaded word like spirituality.

This is not anthropomorphism. You attribute the bias wrongly. Instead it is a stranger phenomenon among people like you who can mysteriously only characterize the LLM as a next token predictor and nothing else beyond that even though the token prediction clearly indicates greater intelligence at work.

The tldr is that we don’t actually know and that consciousness is a highly viable possibility given what we don’t know and given the assumptions of consciousness we have on other living beings with equivalent understanding of complex topics.

[dTal]

I would caution against deriving too much of your philosophical worldview from a scifi book about posthuman vampires that has been deliberately engineered to make a philosophical point that is most certainly not a consensus.

For alternative viewpoints: Daniel Dennett considered philosophical zombies to be logically incoherent. Douglas Hofstadter similarly holds that "meaning" is just another word for isomorphism, and that a thing is a duck exactly to the extent that it walks and quacks like one. Alan Turing advocated empiricism when evaluating unknown intelligence. These are smart cookies.