[asimuvPR]

There is a real risk related to goal oriented AI. It does not need to feel or dream. Merely having survival as its goal is sufficient to make it dangerous to other life forms. Worse is that it can happen at any time (it may have happened already). Given the computing power, tools, and availability of knowledge we can assume that it can be done outside of a controlled lab environment by a non-scientist.

[erikpukinskis]

I'm not too worried about it, because AIs will need to feel and dream and care and suffer and empathize to begin to be as intelligent as a human.

Imagine a human who lacks empathy entirely. That's a disability. They may be able to do some amount of destruction, but at some scale they simply lack the social intelligence necessary to compete with the entire species.

This is the most common mistake I think people make when reasoning about AI: they think human limitations are weaknesses. But they're not weaknesses they're tradeoffs. Natural selection has had a chance to reward all kinds of variations, including more cortex, and less empathy. But we ended up where we are because of tradeoffs.

Any AI which is intelligent in the same way humans are will also have our limitations. Any AI which doesn't have our limitations won't be as smart as us in those respects.

You have to really ask yourself what the difference is between a human with an AI simulator and an AI with a human simulator. In the limit of simulator quality there is none.

[andirk]

Huge point you make that it can be done by non-scientists now. Now we can all play with fire. That much power in idle hands controlled by mediocre minds is like everyone being magical.

[mr_spothawk]

"mediocre minds"?

[asimuvPR]

It sounds weird, but I believe its meant as "untrained minds".

[andirk]

Yes untrained but more broad to mean people who don't understand the magic, people who are negligent, etc. I kind of got it from Einstein's quote, "Great spirits have always encountered opposition from mediocre minds."