[rjinman]

> What's the difference between your "agentic AIs" and, say, "script kiddies" or "expert anarchist/black-hat hackers"?

Intelligence. I'm talking about super-intelligence. If you want to know what it feels like to be intellectually outclassed by a machine, download the latest Go engine and have fun losing again and again while not understanding why. Now imagine an ASI that isn't confined to the Go board, but operating out in the world. It's doing things you don't like at speeds you can scarcely comprehend and there's not a thing you can do about it.

[semi-extrinsic]

But the world is not a game where you "win" by intelligence; very far from it. Just look at who is currently in the White House.

[tmiku]

> Now imagine an ASI that isn't confined to the Go board, but operating out in the world.

I don't think it's reasonable at all to look at a system's capability in games with perfect and easily-ingested information and extrapolate about its future capabilities interacting with the real world. What makes you confident that these problem domains are compatible?

[rjinman]

That’s not what I was saying at all. I was using Go as an example of what the experience of being helplessly outclassed by a superior intelligence is like: you are losing and you don’t know why and there’s nothing you can do.

[rybosworld]

I completely agree with you. Chess/Go/Poker have shown that these systems can become so advanced, it becomes impossible for a human to understand why the AI chose a move.

Talk to the best chess players in the world and they'll tell you flat out they can't begin to understand some of the engine's moves.

It won't be any different with ASI. It will do things for reasons we are incapable of understanding. Some of those things, will certainly be harmful to humans.