[personlurking]

Right, but what I wonder is why we would need something that already does what we want without intervention to become even smarter (leading to it possibly becoming all-powerful)? What are the real-world advantages of making it self-aware?

[em3rgent0rdr]

Unassisted learning will unleash great advances in computing. Imagine not having to write a program to accomplish some task, but rather just putting some ai in an environment with some obstacle and letting it figure out how to teach itself to overcome it. Advantage: no more need for humans to have to program. Disadvantage: self-aware, all-powerful AI.

[coldtea]

>Unassisted learning will unleash great advances in computing. (...) Advantage: no more need for humans to have to program. Disadvantage: self-aware, all-powerful AI.

So it's like "nuclear bombs will kill all pesky mosquitos. Disadvantage: humanity wiped out too".