| Elon Musk had a thought provoking take on this question. I'll roughly summarize, and add some nuances he didn't mention. He was speaking to a room full of state governors, most of whom seemed fairly unconvinced by his concerns. He said that he didn't believe his example was real, but that it was illustrative. Consider an AI whose goal is to maximize stock market returns. So this AI ingests vast sums of historical data, of all kinds, and it considers https://en.wikipedia.org/wiki/Korean_Air_Lines_Flight_007 where the Soviet Union shot down a Korean Air Lines 747 in 1983 when it deviated slightly from its path from Anchorage to Seoul and entered the edge of Soviet protected airspace. About the same time, there was a nearby US aerial reconnaissance mission. This event greatly increased tensions during some of the worst years of the cold war, and as a result, certain kinds of stock market issues moved in certain directions. Considering this incident, and the state of the world in 2014, the AI takes three actions: 1. Buys/shorts the same and/or related stocks, as applicable. 2. Creates some fake intelligence chatter, consumed by the Russians, that where was going to be an areal target of interest over east Ukraine on a certain day and time. The location and direction would not clearly match known commercial airline flights. 3. Break into the flight computer of Malaysia Airlines Flight 17 on 17-July-2014, and cause it to fly slightly off course. https://en.wikipedia.org/wiki/Malaysia_Airlines_Flight_17 The result? Another civilian airliner was shot down by the Russians, which increased global political tensions, and probably caused some stocks to move in relatively predictable ways. In this example, the AI met the goal it was given in an exceedingly unexpected and unpleasant way. This is clearly NOT an end of the world scenario, and is a hypothetical. But it is, in my mind, illustrative of some of the dangers of AI. Systems that can do powerful, unpredictable things in order to meet mundane and reasonable goals. |