[qualifiedai]

"Accidental intelligence explosion" - you have to provide a reasonable argument that this can happen. AIs we have now or currently under development are still just tools in the sense that all agency and consciousness comes solely from human operators. Of course, human operators can be malicious, which is why we should regulate applications of AI, not fundamental research in it.

[Strilanc]

I'm sure you're familiar with the arguments. It's the explicit goal of several AI companies to make something more capable than us at engineering. Once you have that, there's the possibility of very rapid self-reinforcing improvement. If you lose control of something like that, it's game over.

GPT4 may not generate world class code, but it does it at a scale and speed unmatched by humanity. Alpha Zero took a week to go from nothing to better than any human in history at Go.