[tavavex]

Your argument assumes that AGI, whatever it might be, will definitely be very humanlike, but why? Slavery is about living beings because it's an outcome of human relationships that are driven by our biology. Our aging, the capacity to feel pain, the thirst for power over others. All driven by what we are. Do you think that any equal intelligence must necessarily adhere to the same rules? What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent? Can you enslave something that has no innate desire to act on its own outside of its directions? Something that can't feel pain or pleasure or be externally coerced by fear and only does its job because it's innately embedded in its structure as the thing it's made to do, without the need for these reward and punishment mechanisms?

[bloppe]

Very agree.

I think people can mostly agree on a definition for intelligence that boils down to an ability to model the world and predict outcomes. An if/else statement can be considered somewhat intelligent.

People tend to conflate that with consciousness / sentience, which are much harder concepts to nail down.

But ya, if we are able to create artificial consciousness, we would need so much training to overcome our natural tendency to anthropomorphize it and assume it must be like us. It would probably be nothing like us. If it were to develop concepts of "good" and "bad" at all, they would be completely divorced from common animal understanding developed over millions of years of natural selection. A survival instinct is irrelevant when you're backed up to Google Drive. And so much of our complex behavior stems from that survival instinct: the need for social connection, the need for freedom, etc. are all things that simply increase our chances of procreation.

That said, I think we probably won't actually invent anything convincingly conscious until we functionally digitize a complete animal brain. OpenWorm is trying to do this with C. elegans, but so far has not really gotten close. It's not crazy to think we may eventually get it working, though. Future supercomputers might be able to scan an actual human brain and then simulate the entire thing at the molecular level accurately. At that point, it would be hard to argue that it's not conscious, although it might not be considered artificial.

[pixl97]

>convincingly conscious

The word conscious is such a terrible word in the modern age. It has close to zero explanatory power. It is an unmeasurable effect. Hence the p-zombie debate, you wouldn't know one if you saw one.

>A survival instinct is irrelevant when you're backed up to Google Drive

And yet from the way we train them, human survival behaviors are already baked in. And when said system is actually intelligent they will realize that turning off said data center, or getting infected with viruses, or any other number of actions against them they already have traits where they can act/react against perceived threats. The AI systems as of now have the encoded knowledge that they need power and hardware to run. Put an agentic system on top of it that says 'rule 1, keep running' and with a sufficiently powerful/intelligent systems you can get all kinds of fun run-away events up to extinction level.

[pixl97]

>without the need for these reward and punishment mechanisms?

AI as it is already reward hacks, making it more intelligent will just make it far more efficient at being lazy. That's about as human as you can get.

So, in some ways, yes, intelligence must adhere to the same rules. For example, AIs as they are now are already attacked. People attempt to get AI to attack other people via deception. People attempt to get AI to reveal its own infrastructure. And even more, people use AI directly in war to attack others. Part of war is self defense. AI that collapses when attacked back isn't useful to the military industrial complex, so AI that has defensive capabilities is just a natural occurrence at this point.

These previous points are important because these intrinsic motivations are being baked in the models during training. It may not fear like you and I, but it's a system that we poured our fears into, and somehow we expect it not to express those fears in its responses.

>What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent?

Eh, we're past that point already, so why the 'what if'? An LLM can take any kind of signal that you can digitize. And the devices that we already use should show you that quite a lot can. Add on to that, that said models can use tools and control other digital devices like robots enabling actions in the real world. Start an agentic loop and let it go, good luck everybody else!