[matthewtromp]

I don't really see why ASI wouldn't be able to dominate the world of atoms. All it needs is some humanoid robots, or even just humans willing to do what it says. That seems pretty achievable to me! We're already working on humanoid robots.

Fable is pretty smart, yeah! That's one of the things that concerns me. So far it seems like its intelligence isn't totally general. It's great at coding, and great at finding connections between facts, and great at making/critiquing arguments, but maybe it's not quite great at e.g. making complicated strategies to take over the world. But I think there's good reasons to believe that, as models get smarter and smarter, they will get better and better at stuff like that. And that worries me!

The history of AI is littered with people saying "yeah sure, AI can do X, but it will never be able to do Y, because Y is a fundamental part of the human condition." We're running out of Ys in this sentence!