[cosmodisk]

What I miss most,in discussions about AI, is the motivation factor,which is the driving force behind every single thing we humans do. How can we create a system that would be motivated to evolve in order to better itself. Humans created all sorts of things because fear,hunger,or pleasure was so strong and it couldn't be pushed away. What will happen to an AI powered robot that one day decide that going into radioactive areas isn't quite what it wants and will say 'screw it'?

[simonh]

The issue is that intelligence by itself provides to motivations or objectives. Intelligence is simply a tool to be used in order to get things done. In humans the objectives are provided by instincts, emotions and biological needs.

In AI we will have to provide the goals, but as the paper clip maximiser thought experiment shows, we’re going to have to be very careful and thoughtful about it.