[Geee]

The pessimistic case is based on the misconception that AI is some kind of a superhuman. Our current AI models are trained on human data, which has an unfortunate side effect which causes them to think and behave like a human. But as soon as we learn to train them without human data, we find out that AI is just a supercalculator, and it won't have any own will or agency.

Will and agency are primal biological instincts, which a pure intelligence doesn't have. It doesn't want or need anything. Therefore it won't act.

A superintelligence with human primal instincts would be scary indeed, but obviously we don't want to build that.

[iamsaitam]

What you mention of training without human data seems to me an impossibility. Unless you're talking about going back to programming an AI via traditional methods rather than relying on machine learning (which might not be impossible, hard to prove it as such at least).

I don't think you can divorce intelligence from all biological aspects and just get computational power. It's an interesting question though..

[ianm218]

Increasingly AI's are trained using reinforcement learning [1] so these are not really human tasks but things like trying to prove theorems, play games, solve code problems and getting feedback from compilers and similar. A lot of the early pop science coverage of AI was around the ideas of "data walls" and constrains of human data, most of which just wasn't really true or long term true anyway.

[1]. https://www.dwarkesh.com/p/dario-amodei-2