| If anyone is really worried about magic AI, they can build it inside a VR bottle and add failsafes to protect against god-like supersentience. I think god-like supersentience is ludicrously unlikely, if only because it would probably have to be NP complete and bug free, and both seem like a very tall order - you immediately run into the barber's paradox where you're expecting a system to understand itself completely using its own system. Is this likely? I'd say no. In fact I conjecture it's physically and metaphysically impossible. More mundane super-AIs are possible, but they reduce to the out of control machine problem.If there's a problem, it will be because of the possible destructiveness of the implementation technology - e.g. nano, or even traditional bio - and not inherently because of AI. Bio-like systems may turn out to be more efficient than silicon, so we're far more likely to have a problem if we build a smart machine self-evolving machine out of DNA than out of GPUs. Arguably we're a smart self-evolving machine already, and the jury is still out on whether or not we're a good idea. |
That's crazy. Human brains were created by nothing more than random mutations and selection. A very stupid algorithm created the most intelligent thing that currently exists. Humans are a thousand times smarter than evolution. We can invent things evolution never could.
And now humans are learning about how our own brains work through studying them, and also how to build intelligent algorithms ourselves. And we are doing it much faster than evolution did (10's of years vs millions.)
>the barber's paradox where you're expecting a system to understand itself completely using its own system.
I don't see how this is a paradox. There's no reason a system can't understand itself. The algorithms in your brain are probably much simpler than the amount of space your brain has to store information.
>Bio-like systems may turn out to be more efficient than silicon
Silicon is already vastly more efficient. Transistors can be the width of atoms, while synapses are the widths of many molecules. Neurons have to maintain all the machinery necessary for life and self replication. They use slow and inefficient chemical signals and reactions. Etc, etc.