[someguyorother]

There are ways to keep an AI in sealed hardware and making sure it can't affect the world, for instance by using an objective function that only deals with mathematics, and doesn't deal with the real world at all.

E.g. the AI is given a fixed amount of hardware and told to produce an algorithm that solves some NP-complete problem (say integer programming) in expected time as close to polytime as possible, as well as a mathematical proof that the algorithm satisfies the claimed close-to-polytime complexity bound. Then humanity can just solve the NP-complete problems separately once they have the algorithm.

This objective function doesn't care about the physical world -- it doesn't even know that a physical world exist -- and so it's about as likely to directly affect the physical world as MCTS or AlphaGo.

The "AI is going to run out of control" is a very compelling narrative (as everybody who has read the Sorcerer's Apprentice understands). But that doesn't make it true. Beware the availability heuristic.

(Incidentally, I think AI destroying mankind because it's too smart is an unlikely outcome. It's much easier for the AI to subvert the human-designed sensors linked to its objective function; and if the AI is sufficiently smart and the sensors aren't perfect, then it can always do so.)

[mhb]

These counterarguments are only possibly effective because you're imagining some particular kind of AI. When there is a useful AI, of course we will want it to be able to interact with people and have it control physical things in the real world. Just like existing computers do.