[agronomov]

Well, imagine advanced AI beyond imagination. It is so advanced that it mimicks all of the human activity in exactly the same way. By definition it would be able to do whatever we are doing (and probably better)

Unless you don't believe that such an AI is possible, in which case I wonder if you have tried GPT-4 and whether you remember what chat bots were like 5 years ago!

This of course would be AGI. It would still not be conscious though, but a lot of people would fall victim to it and believe that it is. And that's what will end us. We will not pass the mirror test (yep, this one https://en.wikipedia.org/wiki/Mirror_test)

[bell-cot]

In a Philosophy 307 class, your "advanced AI beyond imagination...mimic all of the human activity..." description begs some questions about souls, or other "special sparks of magic" which somehow distinguish such AI's from real humans. And exactly how those souls or sparks or whatever might pass Carl Sagan's Invisible Dragon Test.

But in the real world - those same descriptive words are written by a Marketing Dept. The actual engineers, who developed and built the AI, on an all-too-finite budget, know otherwise. And know that the PHB's already have plans to cut the production cost - by collecting data on which parts of the already-limited "mimic all human activity" are ignored by the AI's actual customers, and quietly nerfing or removing those parts.

[agronomov]

That special spark of magic is my feelings that I experience all the time. I'm not talking about souls -- amoeba probably also experiences things (just by analogy form us)