[alehul]

> Further, the second a smart one starts getting devastating results, there will be political push-back to do something about it. People might cast votes to make rules about them that limit their power or rate of development.

My fear of AGI is that it will not be able to be stopped once deployed, as you're saying. It's irreversible. It will know that we want to shut it down, and so it will be able to copy itself onto other devices (think about the hacking capabilities of AGI for a moment), or any other method of survival.

The current approach is to regulate it after it is invented. While this worked for cars and planes and many other inventions, AGI is different for the reason above.

In fact, with that in mind, a scary thought is that any group of researchers who are cognizant of that would hide their creation of AGI if successful, assuming they were motivated by profit. Thus it would remain woefully unregulated.

[throwawaymath]

> think about the hacking capabilities of AGI for a moment

We're discussing the hypothetical skills and capabilities of a thing which is fundamentally science fiction. The rules are treated as arbitrary.

I don't see a priori why an AGI would be intrinsically good at hacking, or even why it would be capable of exponentially improving itself.

This is the problem I see with any discussion of AGI. The game's rules don't matter, so we can define whichever properties we'd like for the sake of argument. There's no skin in the game to counteract that, because we have no conception of what AGI will actually be like - nor if it's even possible.

As it stands, in your comment and the rest of this thread I see a variety of leaps and jumps to scenarios which seem completely undefended.

[nickpsecurity]

I try to stay grounded by comparing them to both prior AI's, regular brains, savant intelligence, and combinations of them playing to each's strengths. When I do this, I realize that we already have systems in place preventing, reducing, and containing damage from all kinds of smart humans. We react to their schemes in areas like finance in cat and mouse game. The AGI's would probably be no worse than dealing with them so long as they're designed to be hard or impossible to copy themselves.