[reallyeli]

> Luminaries in the field such as Demis Hassabis and Yann LeCun believe that AI can turbocharge scientific progress and lead to a golden age of discovery. Could they be right?... Such claims are worth examining, and may provide a useful counterbalance to fears about large-scale unemployment and killer robots.

But will we maintain control of the above-human-ability, autonomous AI systems these companies are racing to build? This is the AI control problem.

If not, then "AI can automate science" isn't much of a counterpoint or reason to be optimistic -- science may be automated, but not under any human's control and not for any human's benefit. In fact, if we're in this situation, the ability of AI systems to automate science is worse news than otherwise, in the same way that the invention of science by humans was bad (or at best, very mixed) news for the animals of Earth.

[Vecr]

I'm pretty sure Yann LeCun at one point didn't really care if AIs replaced humans, but I think people got through to him that what happens after AIs take over would almost certainly look really boring to a hypothetical intelligent human observer, even assuming the AI system survives into the long term. In the set of possible AIs that kill all humans, I'd suggest that almost all of them are not properly aligned to their own long-term survival either.