In real life it won’t be that the computer prevents you from turning it off. It’ll be that the computer is guarded by cultists who think its god, and unstoppable market forces that require it to keep running.
When AI ends up running everything essential to survival and society, it’ll be preposterous to even suggest pulling the plug just because it does something bad.
Can you imagine the chaos of completely turning off GPS or Gmail today? Now imagine pulling the plug on something in the near future that controls all electric power distribution, banking communications, and Internet routing.
This is the case with capitalism today. I don't like where he took the philosophy, but Nick Land did have an insight that all the worst things we believe about AI (e.g. paperclip optimizing etc) are capitalism in a nutshell.
Just listen to what these CEOs say on the topic and they basically admit something terrible is being built, but that the most important things is that they are the ones to do it first.
Can you imagine the chaos of completely turning off GPS or Gmail today? Now imagine pulling the plug on something in the near future that controls all electric power distribution, banking communications, and Internet routing.