| The fundamental problem is that we keep setting 'reasonable' benchmarks, implicitly because we want to make them seem reachable, but then when we inevitably do we find it feels we [of course] haven't achieved what we really wanted to in the first place. True artificial intelligence would have the ability to meaningfully and significantly improve itself, recursively - a characteristic of even the most primitive life on Earth. And the implications of a computing system capable of such simply can't be overstated. With access to essentially all contemporary knowledge, flawless recall, and stupidly ridiculous amounts of energy to power itself (relative to e.g. a brain), it would create an explosion of knowledge in every single domain imaginable, and many we can't even yet fathom. And it would be doing this at an accelerating pace owing to self improvement. Basically the very nature of existence and knowledge would change. Sentience or not would be irrelevant as we simply tried (and inevitably failed, miserably) to catch up. But this is a bit further out of reach than 'can vaguely pass for a human in a casual non-adversarial q&a style chat.' --- This was one of the rather many areas where Star Trek failed to really consider the implications of its concepts, probably because it would simply break the world building. The Borg, for instance, would be defacto Gods in no time, even without assimilation. |
This is true of pretty much all scifi! It's funny seeing super-futuristic depictions of star-fighter pilots and combatants with firearms and its just...so crushingly evident that humans will not have supremacy in these arenas very shortly.
Frank Herbert must have anticipated this complication and side-stepped the whole issue by preemptively canonizing the Butlerian Jihad.