[mikewarot]

Having finished the episode, it seems quite clear to me that Lex just doesn't understand the argument, or doesn't want to understand. He's so used to the idea of falling in love with an AI that he can't see the danger.

I see the danger, let me give an analogy.

What if, according to the laws of physics, it were possible to make a thermonuclear weapon out of beach sand using a microwave oven.

That's something so absurd that we'd never figure it out, but AGI could. That scale of dangerously destabilizing knowledge could show up at any time from a superintelligent AGI.

Its bad enough that nation-states have the resources to make civilization ending weapons. I think AGI could super-empower those with access to it.

---

On the other hand, what if it were possible to make unlimited clean energy using beach sand, a microwave oven and some whiskey as a catalyst. AGI could make that future possible as well.