[jameshart]

First: what do you mean 'upgrade everything from human secure'? I think if we've learnt anything recently it's that basically nothing is currently even human secure, let alone superintelligent AI secure.

Second: most doomsday scenarios around superintelligent AI are, I suspect, promulgated by software guys (or philosophers, who are more mindware guys). It assumes the hardware layer is easy for the AI to interface with. Manufacturing nanites, bioengineering pathogens, or whatever other WMD you want to imagine the AI deciding to create, would require raw materials, capital infrastructure, energy. These are not things software can just magic up, they have to come from somewhere. They are constrained by the laws of physics. It's not like half an hour after you create superintelligent AI, suddenly you're up to your neck in gray goo.

Third: any superintelligent AI, the moment it begins to reflect upon itself and attempt to investigate how it itself works, is going to cause itself to buffer overrun or smash its own stack and crash. This is the main reason why we should continue to build critical software using memory unsafe languages like C.

[Micaiah_Chang]

By 'upgrade everything from human secure' I meant that some targets aren't necessarily appealing to human targets but would be for AI targets. For example, for the vast majority of people, it's not worthwhile to hack medical devices or refrigerators, there's just no money or advantage in it. But for an AI who could be throttled by computational speed or wishes people harm, they would be an appealing target. There just isn't any incentive for those things to be secured at all unless everyone takes this threat seriously.

I don't understand how you arrived at point 3. Are you claiming that somehow memory safety is impossible, even for human level actors? Or that the AI somehow can't reason about memory safety? Or that it's impossible to have self reflection in C? All of these seem like supremely uncharitable interpretations. Help me out here.

Even ignoring that, there's nothing preventing the AI from creating another AI with the same/similar goals and abdicating to its decisions.

[jameshart]

My point 3 was, somewhat snarkily, that AI will be built by humans on a foundation of crappy software, riddled with bugs, and that therefore it would very likely wind up crashing itself.

I am not a techno-optimist.