[ChrisKnott]

> "It’s like imagining that a printer could actually feel pain because it can print bumper stickers with the words ‘Baby don’t hurt me’ on them. It doesn’t matter if the next version of the printer can print out those stickers faster, or if it can format the text in bold red capital letters instead of small black ones. Those are indicators that you have a more capable printer but not indicators that it is any closer to actually feeling anything"

Love TC but I don't think this argument holds water. You need to really get into the weeds of what "actually feeling" means.

To use a TC-style example... suppose it's a major political issue in the future about AI-rights and whether AIs "really" think and "really" feel the things they claim. Eventually we invent an fMRI machine and model of the brain that can conclusively explain the difference between what "really" feeling is, and only pretending. We actually know exactly which gene sequence is responsible for real intelligence. Here's the twist... it turns out 20% of humans don't have it. The fake intelligences have lived among us for millennia...!

[epr]

I disagree. The reason humans anthropomorphize "AI" is because we apply our own meta-models of intelligence to llms, etc., where they simply don't apply. The model can spit out something that seems extremely intelligent and well thought out that would truly be shocking if a monkey said it for example due to our meta-model of intelligence, and that may be valid in that case if we determined it wasn't simply memorized. His argument can certainly be more fleshed out, but the point he's making is correct, which is that we can't treat the output of a machine designed to replicate human input as though it contains the requisite intelligence/"feeling"/etc to produce that output on it's own.

[ChrisKnott]

I agree that with current LLMs the error goes the other way; they appear more conscious than they are, compared to, say, crows or octopuses which appear less conscious than they actually are.

My point is that "appears conscious" is really the only test there is. In what way is a human that says "that hurts" really feeling pain? What about Stephen Hawking "saying it", what about if he could only communicate through printed paper etc etc. You can always play this dial-down-the-consciousness game.

People used to say fish don't feel pain, they are "merely responding to stimulus".

[mewpmewp2]

The only actual difference in my view is that somehow we feel that we are so uber special. Besides that, it seems there's no reason to believe that we are anything more than chemical signals. But the fact that we have this strong "feeling" that we are special refuses us to admit that. I feel like I'm special, I feel like I exist. That's the only argument for being more than something else.

[asdf6969]

> I feel like I exist.

This is pretty much the only thing in the world that you can definitely prove to yourself. It’s not a feeling

[pixl97]

Hell, people used to say other people of different races don't feel pain, so we're not a great group to ask because of our biases and motivations.

[jyounker]

I'm pretty sure that there are still people around who believe that to some degree.

[gcanyon]

Interestingly the movie Companion, out this weekend, illustrates this case exactly. It's a thriller, not a philosophical treatise, so don't expect it to go deep into the subject, but the question of what "pain" means to an AI is definitely part of the story.

[ge96]

I like the one with Ryan Gosling more

[thwackamole]

You appear to be conflating 'feeling' and 'intelligence', which is not what TC is doing.

He is also not wrong about whether current AIs experience feelings. I suggest you learn more about the neuroscience of feelings.

[ChrisKnott]

Well, he is making an analogy that real internal experience cannot be confirmed externally, however convincing the performance, but this is the only way we know about the internal experience of all things, including ones we typically assign "real" consciousness to (humans, dogs) and ones we don't (amoeba, zygotes, LLMs).

To be clear I'm not for a moment suggesting current AIs are remotely comparable to animals.

[asdf6969]

> You need to really get into the weeds of what "actually feeling" means.

We don’t even know what this means when it’s applied to humans. We could explain what it looks like in the brain but we don’t know what causes the perception itself. Unless you think a perfect digital replica of a brain could have an inner sense of existence

Since we don’t know what “feeling” actually is there’s no evidence either way that a computer can do it. I will never believe it’s possible for an LLM to feel.

[layer8]

> I will never believe it’s possible for an LLM to feel.

Why is that, given that, as you state, we don’t know what “feeling” actually is?

[armchairhacker]

“Feeling” is disconnected from reality, it’s whatever you perceive it as. Like morality, you can’t disprove someone’s definition of feeling, you can only disagree with it.

If scientists invent a way to measure “feeling” that states 20% of people don’t feel, including those otherwise indistinguishable from feeling ones, most people would disagree with the measurement. Similarly, most people would disagree that a printer that prints “baby don’t hurt me” is truly in pain.

[nonameiguess]

These discussions seem to me to still get hung up on the classical sci-fi view of an AI, even talking about Companion here, of some single identifiable discrete entity that can even potentially be the locus of things like rights and feelings.

What is ChatGPT? Ollama? DeepSeek-R1? They're software. Software is a file. It's a sequence of bytes that can be loaded into memory, with the code portion pulled into a processor to tell it what to do. Between instructions, the operating system it runs on context switches it out back to memory, possibly to disk. Possibly it may crash in the middle of an instruction, but if the prior state was stored off somewhere, it can be recovered.

When you interact through a web API, what are you actually interacting with? There may be thousands of servers striped across the planet constantly being brought offline and online for maintenance, upgrades, A/B tests, hardware decommissioning. The fact that the context window and chat history is stored out of band from the software itself provides an illusion that you're talking to some continually existing individual thing, but you're not. Every individual request may be served by a separate ephemeral process that exists long enough to serve that request and then never exists again.

What is doing the "feeling" here? The processor? Whole server? The collection? The entire Internet? When is it feeling? In the 3 out of 30000 time slices per microsecond that the instruction executing is one pulled from ChatGPT and not the 190 other processes running at the same time that weren't created by machine learning and don't produce output that a human would look at and might think a human produced it?

I'll admit that humans are also pretty mysterious if you reduce us to the unit of computation and most of what goes on in the body and brain has nothing to do with either feeling or cognition, but we know at least there is some qualitative, categorical difference at the structural level between us and sponges. We didn't just get a software upgrade. A GPU running ChatGPT, on the other hand, is exactly the same as a GPU running Minecraft. Why would a fMRI looking at one versus the other see a difference? It's executing the same instructions, possibly even acting on virtually if not totally identical byte streams, and it's only at a higher-level step of encoding that an output device interprets one as rasters and one as characters. You could obfuscate the code the way malware does to hide itself, totally changing the magnetic signature, but produce exactly the same output.

Consider where that leads as a thought experiment. Remove the text encodings from all of the computers involved, or just remove all input validation and feed ChatGPT a stream of random bytes. It'll still do the same thing, but it will produce garbage that means nothing. Would you still recognize it as an intelligent, thinking, feeling thing? If a human suffers some injury to eyes and ears, or is sent to a sensory deprivation chamber, we would say yes, they are still a thinking, feeling, intelligent creature. Our ability to produce sound waves that encode information intelligible to others is an important characteristic, but it's not a necessary characteristic. It doesn't define us. In a vacuum as the last person alive with no way to speak and no one to speak to, we'd still be human. In a vacuum as the last server alive with no humans left, ChatGPT would be dirty memory pages never getting used and eventually being written out to disk by its operating system as the server it had been running on performs automated maintenance functions until it hits a scheduled shutdown, runs out of power, or gets thermally throttled by its BIOS because the data center is no longer being actively cooled.

I think Ted Chiang is doing us a service here. Behavioral equivalence with respect to the production of digitally-encoded information is not equivalence. These things are not like us.

[drbig]

> Behavioral equivalence

It seems for a lot of people that's all that matters: "if it quacks like a duck it must be a duck!". I find that short-sighted at best, but it's always difficult to present arguments that would "resonate" with the other side...

[almusdives]

> We actually know exactly which gene sequence is responsible for real intelligence.

We don't at all know this.

[borski]

It’s an idea for a science fiction story.

[bee_rider]

It isn’t just an idea for a science fiction story though. It is also a philosophical argument, just predicated on something unexpected, which is probably not true, but which presents for us an interesting scenario, and which isn’t technically ruled out by existing evidence (although, it seems unlikely of course).

Well, I guess that’s what the best science fiction stories are. But, the best science fiction stories aren’t just science fiction stories!

[jodrellblank]

“suppose”