[batavian]

It doesn't really even sound sci-fi, or rather, it would make for some bad sci-fi. What we are at risk of, is having deterministic model artifacts trigger complex unintended consequences from being fetishized as decision-making agents. Considering how few CS and AI people I've met who even know that e.g. pragmatics are a critical component of sentence meaning, I sense the most proximate AI dystopia is a lot closer to this latter scenario.

Conceptually the possibility of an AGI does present a real challenge (& one worth grappling); in practice, we haven't made any progress really toward realizing that problem. We've become good at feeding "large" datasets into algorithms that are many orders of magnitude less advanced than a lizard brain. Then we get to the issue that our large datasets are only large when thinking of classical statistical sample sizes. They nonetheless are lacking in redundancy, interactivity, stakes or any centered perspective such that there is no chance of anything resembling intelligence, agency, or plain old intentions arising.