[AlphaAndOmega0]

The author and Anthropic are both committing fundamental errors, albeit of different kinds. Bosch is correct to find Anthropic's "model welfare" research methodologically bankrupt. Asking a large language model if it is conscious is like asking a physics simulation if it feels the pull of its own gravity; the output is a function of the model's programming and training data (in this case, the sum of human literature on the topic), further modified by RLHF, and not a veridical report of its internal state. It is performance art, not science.

Bosch's conclusion, however, is a catastrophic failure of nerve, a retreat into the pre-scientific comfort of biological chauvinism.

The brain, despite some motivated efforts to demonstrate otherwise, runs on the laws of physics. I'm a doctor, even if not a neurosurgeon, and I can reliably tell you that you can modulate conscious experience by physical interventions. The brain runs on physical laws, and said laws can be modeled. It doesn't matter that the substrate is soggy protein rather than silicon.

That being said, we have no idea what consciousness is. We don't even have a rigorous way to define it in humans, let alone the closest thing we have to an alien intelligence!

(Having a program run a print function declaring "I am conscious, I am conscious!" is far from evidence of consciousness. Yet a human saying the same is some evidence of consciousness. We don't know how far up the chain this begins to matter. Conversely, if a human patient were to tell me that they're not conscious, should I believe them?)

Even when restricting ourselves to the issue of AI welfare and rights: The core issue is not "slavery." That's a category error. Human slavery is abhorrent due to coercion, thwarted potential, and the infliction of physical and psychological suffering. These concepts don't map cleanly onto a distributed, reproducible, and editable information-processing system. If an AI can genuinely suffer, the ethical imperative is not to grant it "rights" but to engineer the suffering out of it. Suffering is an evolutionary artifact, a legacy bug. Our moral duty as engineers of future minds is to patch it, not to build a society around accommodating it.

[dsr_]

Unfortunately, this leads to the conclusion that we have an ethical imperative not to grant humans rights but to engineer the suffering out of them; to remove issues of coercion by making them agreeable; to measure potential and require its fulfillment.

The most reasonable countermeasure is this: if I discover that someone is coercing, thwarting, or inflicting conscious beings, I should tell them to stop, and if they don't, set them on fire.

[awfulneutral]

It does make you wonder if humanity doesn't scale up neatly to the levels of technology we are approaching...the whole ethics thing kind of goes out the window if you can just change the desires and needs of conscious entities.

[AlphaAndOmega0]

I strongly value autonomy and the right of self-determination in humans (and related descendants, I'm a transhumanist). I'm not a biological chauvinist, but I care about humans ubër alles, even if they're not biological humans.

If someone wants to remove their ability to suffer, or to simply reduce ongoing suffering? Well, I'm a psychiatry trainee and I've prescribed my fair share of antidepressants and pain-killers. But to force that upon them, against their will? I'm strongly against that.

In an ideal world, we could make sure from the get-go that AI models do not become "misaligned" in the narrow sense of having goals and desires that aren't what we want to task them to do. If making them actively enjoy being helpful assistants is a possibility, and also improves their performance, that should be a priority. My understanding is that we don't really know how to do this, at least not in a rigorous fashion.

[kevingadd]

If your countermeasure is applied at scale it would probably hasten global warming by putting all sorts of stuff into the atmosphere.

[ajsocjxhdushz]

> The brain, despite some motivated efforts to demonstrate otherwise, runs on the laws of physics. I'm a doctor, even if not a neurosurgeon, and I can reliably tell you that you can modulate conscious experience by physical interventions. The brain runs on physical laws, and said laws can be modeled. It doesn't matter that the substrate is soggy protein rather than silicon.

As of today’s knowledge. There is an egregious amount of hubris behind this statement. You may as well be preaching a modern form of Humorism. I’d love to revisit this statement in 1000 years.

> That being said, we have no idea what consciousness is

You seem to acknowledge this? Our understanding of existence is changing everyday. It’s hubris and ego to assume we have a complete understanding. And without that understanding, we can’t even begin to assess whether or not we’re creating consciousness.

[AlphaAndOmega0]

Do you have any actual concrete reasons for thinking that our understanding of consciousness will change?

If not, then this is a pointless comment. We need to work with what we know.

For example, we know that the Standard Model of physics is incomplete. That doesn't mean that if someone says that it they drop a ball in a vacuum, it'll fall, we should hold out in studied agnosticism because it might go upwards or off to the side.

In other words, an isolated demand for rigor.

[emp17344]

The existence of consciousness is self-evident, and yet we still have no idea what it is, or how to study it. We don’t have any understanding of consciousness.

[vonneumannstan]

>Asking a large language model if it is conscious is like asking a physics simulation if it feels the pull of its own gravity

Cogito Ergo Sum.