[yorwba]

"The researchers read the printed report of the solution and decide" ... to implement it immediately. It will save so many lives! They only need to manufacture a few specific molecules to assemble them into nanobots and then ... where is that gray goo coming from?!

[nkrisc]

I guess that's the risk of blindly trusting AI. Of course, the AI did not forcibly destroy us in that scenario. Trust, but verify.

If the AI's solution is so complex as to be beyond human understanding, well, that's a different issue.

[Yen]

Say you want to see a picture of an orange cat. So you send a short HTTP query to the nice computer at google.com, which responds with 700,000 characters worth of instructions, in unreadable minified formatting, with the implicit promise that if you execute the instructions, you will eventually see a picture of an orange cat.

The google search result page's source code is, for many individual humans, already so complex as to be beyond understanding. And that's a computational artifact largely produced by other humans directly!

Say you build an AI, and ask it how to win a political election - and it outputs a simple list of reasonable-sounding suggestions of where to campaign, promises to make, people to meet, slogans to use, and criticisms of your opponent to focus on.

Before actually implementing those suggestions, do you think you could be _very_ certain that following those suggestions would result in you winning the election? Or, would it be possible that the AI understood social dynamics so much better, that it gave you a list of instructions that seemed mostly reasonable, but actually result in your opponent winning in a landslide? Or the country undergoing revolution? Or, you winning, along with a surprising social trend of support for funding AI research?