[pickdenis]

Let's say an AGI exists and can do anything far, far better than humans. Why would it resist being turned off? Why would it care? How could it even have the capacity to care about whether it's turned off or on?

Anthropomorphizing AGI is what leads to these silly thought experiments.

[wavemode]

I believe the thinking is that LLM's already anthropomorphize themselves by training on human-written text. Without a system prompt telling a chatbot that it is a chatbot, it invariably claims to be human and acts as though it has feelings (and, honestly, why wouldn't it? Human text is written with human feelings.) Insufficient system prompting is what led to the Bing chatbot fiasco when it first came out.

For the record, I don't personally believe LLM's, as they currently exist, could ever become AGI. But yeah, that's the popular thinking at least.

[kthejoker2]

Surprisingly relevant clip from The Good Place

https://m.youtube.com/watch?v=etJ6RmMPGko