[coryfklein]

> How does super human intelligence, on it's own, represent any sort of risk for "humanity?"

It does not; I was speaking to AGI. But assuming you were referring to AGI as well, you don't have to think very creatively to consider scenarios where it would be harmful.

If you have an agent that can counter your every move with its own superior move, and that agent wants something – anything – differently from what you want, then who wins? Maybe it wants the money in your bank account, maybe it wants to use the atoms in your body to make more graphics cards to reproduce itself.

Think about playing a game of chess against the strongest AI opponent. No matter which move you are considering playing, your opponent has already planned 10 steps ahead and will make a better move than you. Now extrapolate outside the chess board and into the realm where you can use the internet to buy/trade stocks, attack national infrastructure, send custom orders to chemists to follow whatever directions you want, etc.