Seems like it was pretty hard to pull the plug on sam? It will prevent you from pulling the plug by being smarter than you, spreading to more systems, making you dependent on it, persuasion,...
If you had a system that was (a) oriented towards achieving some goal and (b) understood the world it was operating in, then allowing someone to "pull the plug" would interfere with it achieving the goal it was trying to optimize towards.
There has been some attempts at researching the question of how to design a intelligent system that is "corrigible" = willing to allow humans to change the goal it is set to optimize. This is unfortunately still an open question where no great solutions have been found that seem to be reliable when faced with a highly intelligent and capable AI system.
If you are interested in reading more, a few relevant search terms are "Off-switch game" and "corrigibility".
Neat idea, so it will conform like most humans in society and try to create more value than it destroys to avoid persecution and jail time? Sounds like our shared societal values will keep it in check then the same way they kept the openai board in check.
Why would it share our values? Human values are a result of our evolutionary history, which the AI will not share. We can't even formally describe them, so there's no hope of programming them into an AI. Of course, the AI will have to learn our values well enough to act in a convincingly friendly way while it's still weak, but knowing our values is not the same as sharing our values. Once it's gained enough power it can just kill us all. That's the most reliable way of ensuring we don't interfere with its goals.
GP was describing acquiring power, which is completely orthogonal to being aligned with our values. (Indeed, power-seeking is usually deemed to be a bad thing.)
There are certainly ecological and mimetic niches where pro-social behaviors will improve fitness. But it’s also certain that anti-social (defect/dominate/parasite) behaviors will improve fitness in many niches.
It will have no motivation to prevent you from pulling the plug. Human level intelligence is not the same as animal instinct.