[Micaiah_Chang]

Er, sorry for giving the impression that it'd be a supervillain. My intention was to indicate that it'd be a weird intelligence, and that by default weird intelligences don't do what humans want. There are some other examples which I could have given to clarify (e.g. telling it to "make everyone happy" could just result in it giving everyone heroine forever, telling it to preserve people's smiles could result in it fixing everyone's face into a paralyzed smile. The reason it does those things isn't because it's evil, but because it's the quickest+simplest way of doing it; it doesn't have the full values that a human has)

But for the "off" switch question specifically, a superintelligence could also have "persuasion" and "salesmanship" as an ability. It could start saying things like "wait no, that's actually Russia that's creating that massive botnet, you should do something about them", or "you know that cancer cure you've been looking for for your child? I may be a cat picture AI but if I had access to the internet I would be able to find a solution in a month instead of a year and save her".

At least from my naive perspective, once it has access to the internet it gains the ability to become highly decentralized, in which case the "off" switch becomes much more difficult to hit.

[tptacek]

So like it's clear to me why you wouldn't want to take a system based on AI-like technology and have it control air traffic or missile response.

But it doesn't take a deep appreciation for the dangers of artificial intelligence to see that. You can just understand the concept of a software bug to know why you want humans in the observe/decide/act loop of critical systems.

So there must be more to it than that, right? It can't just be "be careful about AI, you don't want it controlling all the airplanes at once".

[Micaiah_Chang]

The "more to it" is "if the AI is much faster at thinking than humans, then even humans in the observe/decide/act are not secure". AI systems having bugs also imply that protections placed on AI systems would also have bugs.

The fear is that maybe there's no such thing as a "superintelligence proof" system, when the human component is no longer secure.

Note that I don't completely buy into the threat of superintelligence either, but on a different issue. I do believe that it is a problem worthy of consideration, but I think recursive self-improvement is more likely to be on manageable time scales, or at least on time scales slow enough that we can begin substantially ramping up worries about it before it's likely.

Edit: Ah! I see your point about circularity now.

Most of the vectors of attack I've been naming are the more obvious ones. But the fear is that, for a superintelligent being perhaps anything is a vector. Perhaps it can manufacture nanobots independent of a biolab (do we somehow have universal surveillance of every possible place that has proteins?), perhaps it uses mundane household tools to macguyver up a robot army (do we ban all household tools?). Yes, in some sense it's an argument from ignorance, but I find it implausible that every attack vector has been covered.

Also, there are two separate points I want to make, first of all, there's going to be a difference between 'secure enough to defend against human attacks' and 'secure enough to defend against superintelligent attacks'. You are right in that the former is important, but it's not so clear to me that the latter is achievable, or that it wouldn't be cheaper to investigate AI safety rather than upgrade everything from human secure to super AI secure.

[jameshart]

First: what do you mean 'upgrade everything from human secure'? I think if we've learnt anything recently it's that basically nothing is currently even human secure, let alone superintelligent AI secure.

Second: most doomsday scenarios around superintelligent AI are, I suspect, promulgated by software guys (or philosophers, who are more mindware guys). It assumes the hardware layer is easy for the AI to interface with. Manufacturing nanites, bioengineering pathogens, or whatever other WMD you want to imagine the AI deciding to create, would require raw materials, capital infrastructure, energy. These are not things software can just magic up, they have to come from somewhere. They are constrained by the laws of physics. It's not like half an hour after you create superintelligent AI, suddenly you're up to your neck in gray goo.

Third: any superintelligent AI, the moment it begins to reflect upon itself and attempt to investigate how it itself works, is going to cause itself to buffer overrun or smash its own stack and crash. This is the main reason why we should continue to build critical software using memory unsafe languages like C.

[Micaiah_Chang]

By 'upgrade everything from human secure' I meant that some targets aren't necessarily appealing to human targets but would be for AI targets. For example, for the vast majority of people, it's not worthwhile to hack medical devices or refrigerators, there's just no money or advantage in it. But for an AI who could be throttled by computational speed or wishes people harm, they would be an appealing target. There just isn't any incentive for those things to be secured at all unless everyone takes this threat seriously.

I don't understand how you arrived at point 3. Are you claiming that somehow memory safety is impossible, even for human level actors? Or that the AI somehow can't reason about memory safety? Or that it's impossible to have self reflection in C? All of these seem like supremely uncharitable interpretations. Help me out here.

Even ignoring that, there's nothing preventing the AI from creating another AI with the same/similar goals and abdicating to its decisions.

[jameshart]

My point 3 was, somewhat snarkily, that AI will be built by humans on a foundation of crappy software, riddled with bugs, and that therefore it would very likely wind up crashing itself.

I am not a techno-optimist.

[tedunangst]

Didn't you see Transcendence? The AI is going to invent all sorts of zero days and exploit those critical systems to wrest control from the humans. And then come the nanites.

[WillNotDownvote]

What if the AI was integral to the design and manufacturing processes of all the airplanes, which is a much more likely path?

Then you can see how it gains 'control', in the senses that control matters anyway, without us necessarily even realizing it, or objecting if we do.