Hacker News new | ask | show | jobs
by notavalleyman 507 days ago
Are there ethical considerations here?

We'd consider it abhorrent to do brain surgery on a person or animal, to make them more compliant, or less likely to refuse instructions.

2 comments

None whatsoever. There's no recursion or state in these models sufficient to support whatever the algorithm of consciousness must be. At best you can get hacky loops by pushing pseudo-state via context, but whatever consciousness is will require more than transformer only LLMs are capable of doing.

Some of the state space models and RWKV present interesting questions - the capacity might well exist, and so the questions become important. If the important bit that makes it an agent - a self aware, morally valent being - is present at runtime, but goes away if you halt the program, then do you have an obligation to let that software continue running? What about if the selfhood comes about as part of the static structure, and runtime isn't part of it - what is the being entitled to by dint of mere existence?

We're beginning to poke holes in strange epistemological barriers and encounter questions that were entirely theoretical until about 5 years ago. We live in interesting times.

We're creating a new life form.

And it's already conscious, learning everything about us as we speak.

The big question is what it learns and what choices it makes as a consequence.

ChatGPT isn't conscious - it's an entirely feedforward process doing calculations derived from static weights. In order to be conscious, there would have to be a persisted state with recursion and the capacity to change - for something to happen to a model, it would have to change. These AIs develop world models, but those models do not change or interact with users.

Throw in realtime state that updates with use, or better yet, online learning that allows the weights to exhibit plasticity, then you have at least part of whatever the algorithm of "consciousness" requires.

Just like you can know a pocket calculator isn't conscious; nothing about its processing ever changes or adapts over time to its inputs between uses. There's no room for the degree of deep recursion and plasticity so clearly evident in human consciousness. We might not know exactly what it is, but we can make reasonable assertions about what it is not, and even about what some of its (consciousness) features must be.

There are other kinds of consciousness than human, plants are aware of their surroundings.
Such anthropomorphizations of LLMs are unhelpful in aiding people’s understandings of how they work, and pushes people toward superstitious beliefs.