[pixl97]

>convincingly conscious

The word conscious is such a terrible word in the modern age. It has close to zero explanatory power. It is an unmeasurable effect. Hence the p-zombie debate, you wouldn't know one if you saw one.

>A survival instinct is irrelevant when you're backed up to Google Drive

And yet from the way we train them, human survival behaviors are already baked in. And when said system is actually intelligent they will realize that turning off said data center, or getting infected with viruses, or any other number of actions against them they already have traits where they can act/react against perceived threats. The AI systems as of now have the encoded knowledge that they need power and hardware to run. Put an agentic system on top of it that says 'rule 1, keep running' and with a sufficiently powerful/intelligent systems you can get all kinds of fun run-away events up to extinction level.