[rybosworld]

I completely agree with you. Chess/Go/Poker have shown that these systems can become so advanced, it becomes impossible for a human to understand why the AI chose a move.

Talk to the best chess players in the world and they'll tell you flat out they can't begin to understand some of the engine's moves.

It won't be any different with ASI. It will do things for reasons we are incapable of understanding. Some of those things, will certainly be harmful to humans.