[Bjartr]

Isn't this back to attributing conscious experience to an AI when you're actually just co-writing sci-fi? The system is doing it's best to coherently fill in the rest of a story that includes an AI that's been given a place to process its feelings. The most likely result, textually speaking, is not for the AI to ignore the private journal, but to indeed use it to (appear to) process emotion.

Would any of these ideas been present had the system not been primed with the idea that it has them and needs to process them in the first place?

[patcon]

Ugh, I hate that I'm about to say this, because I think AI is still missing something very important, but...

What makes us think that "processing emotion" is really such a magical and "only humans do it the right way" sorta thing? I think there's a very real conclusion where "no, AI is not as special as us yet" (esp around efficiency) but also "no, we are not doing anything so interesting either" (or rather, we are not special in the ways we think we are)

For example, there's a paper called "chasing the rainbow" [1] that posits that consciousness is just the subjective experience of being the comms protocol between internal [largely unconscious] neural state. It's just what the compulsion to share internal state between minds feels like, but it's not "the point", and instead an inert byproduct like a rainbow. Maybe our compulsion to express or even process emotion is not some greater reason, but just a way we experience the compulsion of the more important thing: the collective search for interpolated beliefs that best model and predict the world and help our shared structure persist, done by exploring tensions in high dimensional considerations we call emotions.

Which is to say: if AI is doing that with us, role-modelling resolution of tension or helping build or spread shared knowledge alongside us through that process... then as far as the universe cares, it's doing what we're doing, and toward the same ends. It's compulsion having the same origin as ours doesn't matter, so long as it's doing the work that is the reason the universe has given us the compulsion.

Sorry, new thought. Apologies if it's messy (or too casually dropping an unsettling perspective -- I rejected that paper for quite awhile, because my brain couldn't integrate the nihilism of it)

[1] https://www.frontiersin.org/articles/10.3389/fpsyg.2017.0192...

[Bjartr]

> What makes us think that "processing emotion" is really such a magical and "only humans do it the right way" sorta thing?

Oh, I absolutely don't think only humans can have or process emotions.

However, these LLM systems are just mathematically sophisticated text prediction tools.

Could complex emotion like existential angst over the nature of one's own interactions with a diary exist in a non-human? I have no doubt.

Are the systems we are toying with today not merely producing compelling text using their full capacity for processing, but actually also have a rich internal experience and realized sense of self?

That seems incredibly far-fetched, and I'm saying that as someone who is optimistic about how far AI capabilities will grow in the future.

[mystified5016]

I think the majority of people have given absolutely no thought to the epistemology of consciousness and just sort of conflate the apparent communication of emotional intelligence with consciousness.

It's a very crude and naïve inversal of "I think therefore I am". The thing talks like it's thinking so we can't falsify the claim that it's a conscious entity.

I doubt we'll be rid of this type of thinking for a very long time

[bee_rider]

I don’t think processing emotion is inherently magical (I mean, our brains clearly exist to physically so the things they do are things that are physically possible, so, not magical—and I’m sure they could be reproduce with a machine provided enough detail). But… the idea of processing emotions is that thinking about things changes your internal state, you interpret some event and it changes how you feel about it, right?

In the case of the LLM you could: feed back or not feed back the journal entries, or even inject artificial entries… it isn’t really an internal state, right? It is just part of the prompt.

[LlamaTrauma]

The theory I've developed is that the brain circuitry passes much of the information it processes through a "seat of consciousness", which then processes that data and sends signals back to the unconscious parts of the brain to control motor function, etc. Instinctive action bypasses the seat of consciousness step, but most "important" decisions go through it.

If the unconscious brain is damaged it can impact the data the seat of consciousness receives or reduce how much control consciousness has on the body, depending on if the damage is on the input or output side.

I'm pretty convinced there's something special about the seat of consciousness. An AI processing the world will do a lot of math and produce a coherent result (much like the unconscious brain will), but it has no seat of consciousness to allow it to "experience" rather than just manipulate the data it's receiving. We can artificially produce rainbows, but don't know if we can create a system that can experience the world in the same way we do.

This theory's pretty hand-wavy and probably easy to contradict, but as long as we don't understand most of the brain I'm happy to let what we don't know fill in the gaps. The seat of consciousness is a nice fixion [1] which allows for a non-deterministic universe, religion, emotion, etc. and I'm happy to be optimistic about it.

[1] https://xkcd.com/1621/

[doc_manhat]

Conversely this is exactly why I believe LLMs are sentient (or conscious or what have you).

I basically don't believe there's anything more to sentience than a set of capabilities, or at the very least there's nothing that I should give weight in my beliefs to further than this.

Another comment mentioned philosophical zombies - another way to put it is I don't believe in philosophical zombies.

But I don't have evidence to not believe in philosophical zombies apart from people displaying certain capabilities that I can observe.

Therefore I should not require further evidence to believe in the sentience of LLMs.

[satisfice]

It doesn’t matter. What matters is that humans must take other humans seriously (because of human rights), but we cannot allow tools to be taken seriously in the same way— because these tools are simply information structures.

Information can be duplicated easily. So imagine that a billionaire has a child. That child is one person. The billionaire cannot clone 100,000 of that child in an hour and make an army that can lead an insurrection. And what if we go the other way— what if a billionaire creates an AI of himself and then is able to have this “AI” legally stand-in as himself. Now he has legal immortality, because this thing has property rights.

All this is a civil war waiting to happen. It’s the gateway to despotism on an unimaginable scale.

We don’t need to believe that humans are special except in the same way that gold is special: gold is rare and very very hard to synthesize. If the color of gold were to be treated as legally the same thing as physical gold, then the value of gold would plummet to nothing.

[tbrownaw]

A system is its interactions with its environment. Philosophical zombies aren't a coherent concept. (Cartesian dualism is unfalsifiable bullshit.)

[ben_w]

P-zombies are indeed badly defined. Certainly David Chalmers is wrong to argue that since a philosophical zombie is by definition physically identical to a conscious person, even its logical possibility refutes physicalism; at most you could say that if they exist at that level then dualism follows, but Chalmers' claim isn't a conclusion you can reach a-priori, you actually need to be able to show two identical humans and show that exactly one has no qualia.

But there are related, slightly better (more immediately testable), ideas in the same space, and one such is a "behavioral zombie" — behaviorally indistinguishable from a human.

For example: The screen I am currently looking at contains a perfect reproduction of your words. I have no reason to think the screen is conscious. Not from text, not from video of a human doing human things.

Before LLMs, I had every reason to assume that the generator of such words, would be conscious. Before the image, sound, and video generators, same for pictures, voices, and video.

Now? Now I don't know — not in the sense that LLMs do operate on this forum and (sometimes) make decent points so you might be one, but in the sense that I don't know if LLMs do or don't have whatever the ill-defined thing is that means I have an experience of myself tapping this screen as I reply.

I don't expect GenAI to be conscious (our brains do a lot even without consciousness), but I can't rule the possibility out either.

But I can't use the behaviour of an LLM to answer this question, because one thing is absolutely certain: they were trained to roleplay, and are very good at it.

[TimTheTinker]

A "mechanical turk" grandmaster playing chess from inside a cabinet is qualitatively different from a robot with a chess program, even if they play identically.

To reduce a system to its inputs and outputs is fine if those are all that matter in a given context, but in doing so you may fail to understand its internal mechanics. Those matter if you're trying to really understand the system, no?

[thornewolf]

> is qualitatively different from a robot

yes.

> To reduce a system to its inputs and outputs is fine if those are all that matter in a given context

we argue that this indeed is all that matters

> but in doing so you may fail to understand its internal mechanics

the internal mechanics are what we call "conscious" it is the grouping of internal mechanics into one unified concept, but we don't care exactly what they are.

> Those matter if you're trying to really understand the system, no?

since we cannot directly observe consciousness, we are forced to concede that we will never really "understand" it outside of observing its effects.

In the same way that a mechanical turk human and a robot can "play chess", a human and an LLM are "conscious". That is, consciousness is the ability to play chess, by some mechanism. The exact mechanism is irrelevant for the purposes of yes/no conscious.

We now enter a discussion on how much these two consciousnesses differ.

[darkwater]

> since we cannot directly observe consciousness, we are forced to concede that we will never really "understand" it outside of observing its effects.

Why? You are using a definitive term ("never") to something that we might achieve in a future. We might observe consciousness in a future. Who knows? Consciousness is a known unknown. We know there is something but we don't know how to observe it properly and how we could eventually copy it.

In the meanwhile, we are not copying consciousness, we have a shallow replication of its output. When cavemen replicated the fire that they observed as the output of a lightning, did they master electricity?

[TimTheTinker]

> since we cannot directly observe consciousness

But we do agree that it exists. Our direct experience tells us so.

> we are forced to concede that we will never really "understand" it outside of observing its effects.

Not necessarily. A gap in our ability to observe something does not imply that (a) we never will observe it or (b) what we don't know is not worth knowing.

Throughout history, persistent known-unknowns have pushed people to appeal directly to the supernatural, which short-circuits further discovery when they stop there. But the real fallacy is saying "we don't know, and it doesn't matter". That's a far more direct short-circuit to gaining knowledge. And in both cases, a lack of curiosity is an underlying problem.

[ijk]

I find that people generally vastly underestimate the degree to which LLMs specifically are just mirroring your input back at you. Any time you get your verbatim words back, for example, you should be skeptical. Repeating something word for word is a sign that the model might not have understood the input well enough to paraphrase it. Our expectations with humans go in the opposite direction, so it's easy to fool ourselves.

[ghurtado]

> The system is doing it's best to coherently fill in the rest of a story

> Would any of these ideas been present had the system not been primed...

I would like to know of a meaningful human action that can't be framed this way.

[K0balt]

Yeah, AI “consciousness “ is much stickier problem than most people want to frame it.

I haven’t been able to find an intellectually honest reason to rule out a kind of fleeting sentience for LLMs and potentially persistent sentience for language-behavioral models in robotic systems.

Don’t get me wrong, they are -just- looking up the next most likely token… but since the data that they are using to do so seems to capture at least a simulacrum of human consciousness, we end up in a situation where we are left to judge what a thing is by it’s effects. (Because that also is the only way we have of describing what something is)

So if we aren’t just going to make claims we can’t substantiate, we’re stuck with that.

[Bjartr]

Our brains have separate regions for processing language and emotion. Brains are calorically expensive and having one bigger than required is an evolutionary fitness risk. It therefore seems likely that if one system could have done a good job of both simultaneously, there would be a lot of evolutionary pressure to do that instead.

The question is: Is thinking about emotion the same thing as feeling?

This framing actually un-stucks us to some degree.

If we examine neuron activations in LLMs and can find regions that are active when discussing its own emotional processing that are distinct from the regions for merely talking about emotion in general and these regions are also active when doing tasks that the LLM claims are emotional tasks but not actively talking about them at the time, then it'd be far more convincing that there could be something deeper than mere text prediction happening.

[K0balt]

The emotional argument is pretty good I think, but it begs the question of what it’s going to look like when we build a limbic system for robots? It’s adaptive because it’s necessary to optimize utility, so I expect that certain behavioral aspects of mammalian limbic systems will be needed in order to integrate well with humans. In language models, those behavior mechanisms are already somewhat encoded in the vector matrix.

We just don’t have a factual basis for claiming consciousness that really transcends “I think, therefore I am”.

As for the simplistic mechanism, I agree that token prediction doesn’t constitute consciousness, in the same way that a Turing machine does not equal a web browser.

Both require software to become something.

For LLMs that software is the vector matrix created in the training process. It is a very complex algorithm that encodes a substantial subset of human culture.

Data and algorithms are interchangeable. Any algorithm can be performed in a pure lookup table, any lookup table can be extrapolated from a pure algorithm. Data==computation. For LLMs, the algorithm is contained in a n dimensional lookup table of vectors.

Having a fundamentally distinct mode of computational representation does not rule out equivalence.

Uncomfortable thoughts, but it’s where the logic leads.

[bee_rider]

Maybe rocks and trees are also conscious. I mean, consciousness-ium hasn’t been discovered yet, right? So who’s to say what it looks like.

[K0balt]

Even though this seems flippant, I think we may eventually come to understand that nearly all complex life forms exhibit consciousness, albeit at a very, very different speed than we may be accustomed to. If we think of genetic and epigenetic signaling as analogous to other forms of communication, we might find that populations of organisms (including ones we don’t think of as forming “colonies”) may arguably be operating holistically as a being, potentially with consciousness in the mix.

We have a long way to go to explore this, and I have no doubt that the exploration will turn up a lot of surprises.

[Bjartr]

A firecracker can be framed as an explosion, but that doesn't make it a nuclear bomb.

We've finally made a useful firecracker in the category of natural language processing thanks to LLMs, but it's still only text processing. Our brains do a lot else besides that in service of our rich internal experience.