| Reminds me of the old essay by 'Eliezer: "The Hidden Complexity of Wishes". https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden... In it, there is a thought experiment of having an "Outcome Pump", a device that makes your wishes come true without violating laws of physics (not counting the unspecified internals of the device), by essentially running an optimization algorithm on possible futures. As the essay concludes, it's the type of genie for which no wish is safe. The way this relates to AI is by highlighting that even ideas most obvious to all of us, like "get my mother out of that burning building!", or "I want these virtual wolves to get better at eating these virtual sheep", carry incredible amount of complexity curried up in them - they're all expressed in context of our shared value system, patterns of thinking, models of the world. When we try to teach machines to do things for us, all that curried up context gets lost in translation. |