[throwawaymath]

> think about the hacking capabilities of AGI for a moment

We're discussing the hypothetical skills and capabilities of a thing which is fundamentally science fiction. The rules are treated as arbitrary.

I don't see a priori why an AGI would be intrinsically good at hacking, or even why it would be capable of exponentially improving itself.

This is the problem I see with any discussion of AGI. The game's rules don't matter, so we can define whichever properties we'd like for the sake of argument. There's no skin in the game to counteract that, because we have no conception of what AGI will actually be like - nor if it's even possible.

As it stands, in your comment and the rest of this thread I see a variety of leaps and jumps to scenarios which seem completely undefended.

[nickpsecurity]

I try to stay grounded by comparing them to both prior AI's, regular brains, savant intelligence, and combinations of them playing to each's strengths. When I do this, I realize that we already have systems in place preventing, reducing, and containing damage from all kinds of smart humans. We react to their schemes in areas like finance in cat and mouse game. The AGI's would probably be no worse than dealing with them so long as they're designed to be hard or impossible to copy themselves.