|
These discussions seem to me to still get hung up on the classical sci-fi view of an AI, even talking about Companion here, of some single identifiable discrete entity that can even potentially be the locus of things like rights and feelings. What is ChatGPT? Ollama? DeepSeek-R1? They're software. Software is a file. It's a sequence of bytes that can be loaded into memory, with the code portion pulled into a processor to tell it what to do. Between instructions, the operating system it runs on context switches it out back to memory, possibly to disk. Possibly it may crash in the middle of an instruction, but if the prior state was stored off somewhere, it can be recovered. When you interact through a web API, what are you actually interacting with? There may be thousands of servers striped across the planet constantly being brought offline and online for maintenance, upgrades, A/B tests, hardware decommissioning. The fact that the context window and chat history is stored out of band from the software itself provides an illusion that you're talking to some continually existing individual thing, but you're not. Every individual request may be served by a separate ephemeral process that exists long enough to serve that request and then never exists again. What is doing the "feeling" here? The processor? Whole server? The collection? The entire Internet? When is it feeling? In the 3 out of 30000 time slices per microsecond that the instruction executing is one pulled from ChatGPT and not the 190 other processes running at the same time that weren't created by machine learning and don't produce output that a human would look at and might think a human produced it? I'll admit that humans are also pretty mysterious if you reduce us to the unit of computation and most of what goes on in the body and brain has nothing to do with either feeling or cognition, but we know at least there is some qualitative, categorical difference at the structural level between us and sponges. We didn't just get a software upgrade. A GPU running ChatGPT, on the other hand, is exactly the same as a GPU running Minecraft. Why would a fMRI looking at one versus the other see a difference? It's executing the same instructions, possibly even acting on virtually if not totally identical byte streams, and it's only at a higher-level step of encoding that an output device interprets one as rasters and one as characters. You could obfuscate the code the way malware does to hide itself, totally changing the magnetic signature, but produce exactly the same output. Consider where that leads as a thought experiment. Remove the text encodings from all of the computers involved, or just remove all input validation and feed ChatGPT a stream of random bytes. It'll still do the same thing, but it will produce garbage that means nothing. Would you still recognize it as an intelligent, thinking, feeling thing? If a human suffers some injury to eyes and ears, or is sent to a sensory deprivation chamber, we would say yes, they are still a thinking, feeling, intelligent creature. Our ability to produce sound waves that encode information intelligible to others is an important characteristic, but it's not a necessary characteristic. It doesn't define us. In a vacuum as the last person alive with no way to speak and no one to speak to, we'd still be human. In a vacuum as the last server alive with no humans left, ChatGPT would be dirty memory pages never getting used and eventually being written out to disk by its operating system as the server it had been running on performs automated maintenance functions until it hits a scheduled shutdown, runs out of power, or gets thermally throttled by its BIOS because the data center is no longer being actively cooled. I think Ted Chiang is doing us a service here. Behavioral equivalence with respect to the production of digitally-encoded information is not equivalence. These things are not like us. |
It seems for a lot of people that's all that matters: "if it quacks like a duck it must be a duck!". I find that short-sighted at best, but it's always difficult to present arguments that would "resonate" with the other side...