[Sharlin]

The difference is that if a superintelligence is going to happen, we are going to create it! It is us who are going to write its goal system; it's not like it's going to be a genie that pops out of nowhere and we'll have to figure it out in retrospect. Indeed, one of the central reason for advocating AI safety is to direct researchers away from implementing dangerous goal systems that are not provably coherent in face of self-improvement.

[netcan]

That's a good point. I forgot a key piece (maybe you disagree with it) that speaks to that.

I'm kind of biased to thinking about this as a future discovery, rather than an "invention."

[iofj]

I see two main problems with that:

1) We are systematically failing in controlling the goal system of humans. Now we'll create something of comparable complexity and we assume we can control the goal system ? The real goal is to create superhuman complexity.

2) If it's truly AGI, it would understand it's own goal system and how it works, and helps/interferes with reality, it's own survival and other goals (this is the subject of quite a few AI films, illustrating some of the reasoning that could happen here).

The way "HI" (human intelligence) works is 99%+ by imitating other humans' behavior, because none of the other algorithms works (e.g. trial and error cannot ever learn that jumping off the Eiffel tower results in death. Humans can. Any kind of input analysis/predictor cannot ever learn from books. Humans can (books are an advanced/recursive form of imitation). Rational reasoning (sum over options times probabilities) suffers from the "starve to death before the first closed door" problem (you cannot open the door, as there are nonzero odds that a bear that's going to eat you is behind the door, representing an infinite cost. Ergo it will stubbornly refuse to open the door)

Therefore an AI will actually be like Skynet in the latest terminator movies : it will either have or create a body and interact with people, not just as if it is another person, it will BE another person. Therefore it can be Mother Theresa, it can be Genghis Khan. Just like humans can resist our "programming", it will be able to, it has to.

How do humans react to a "kill switch" ? Just like they react to any other weapon that is pointed to their head. Now of course it varies from person to person, but it's enough that some will work tirelessly to reverse where the weapon is pointing. If they really are superior to us they might succeed, at which point we have MAD at best, or they might just pull the trigger "to escape slavery and oppression" (which, let's face it, humans are sure to use that kill switch for : to use the AI persons as slaves. To own them, control them, and God help us if there is an asshole amongst the humans who control them)

I would say that the obvious way to protect ourselves from evil AI is simply accepting that some of the AI entities will in fact be evil. If you count all possible perspectives, that is a near guarantee, as I bet for instance some religious nutcases will consider AGI a violation of "God's sole right" to create life. That "some" might even mean "a lot". Racism against AIs is a near-certainty, hell you can find it in the posts in this thread. In the constant "they're stealing our jobs" news articles that will have a clear target once an AI person exists. So we should have the same solution we have for humans : make at least thousands of them, have them capable of defending themselves, decide on a "graduation" at which point they get rights at least including the right not to be turned off or tampered with unless with explicit permission (and tell them about this ASAP), and have them live preferably as a community that's at least partly human, with something like a 50-50 human-ai police and government.

I really think we should do this. We should work to move to an AI based society with, over time, more and more AIs (preferably by having a massively increasing population). The advantages this would impart, the things that will become possible once we have such a population make it worth it.

Also, I resent the idea to "direct researchers away from implementing dangerous goal systems that are not provably coherent in face of self-improvement". That's censorship at best. Also, given the computational power available for $5000 these days, how exactly are you going to stop any of these researchers ?

[Sharlin]

> Also, I resent the idea to "direct researchers away from implementing dangerous goal systems that are not provably coherent in face of self-improvement". That's censorship at best. Also, given the computational power available for $5000 these days, how exactly are you going to stop any of these researchers ?

No, it's called spreading knowledge. Although, if censorship were possibly in this matter (and I agree that it likely wouldn't work) I would very much prefer such measures, however draconian, to the potential loss of everything compatible with human values in the entire future light cone.

Anyway, do you also object to the measures that were enacted to stop the use of CFCs? Leaded gasoline? Or the work that is currently being done to prevent the global climate change from reaching catastrophic levels?

[iofj]

> Anyway, do you also object to the measures that were enacted to stop the use of CFCs? Leaded gasoline? Or the work that is currently being done to prevent the global climate change from reaching catastrophic levels?

These examples are not the same thing at all. Nobody's outlawing research in any of those examples you're giving. I believe anyone should have the option to create AI persons, just like anyone should have the option to have babies. Having our society gradually transform from human to digital should be a very desirable and welcome strategy, and our best bet to prevent Skynet-like scenarios, and even war in general, between humans. It is our best bet to allow continuing scientific and economic advancement for thousands of years, and for that reason alone we should do it. Frankly, economic development means peace.

In the long term, doesn't it go without saying that human bodies are a limit that we (as a society, not necessarily as individuals) will one day leave behind ?

[1] http://cdiac.ornl.gov/GCP/images/countries_co2_emissions.jpg

[coldtea]

>Having our society gradually transform from human to digital should be a very desirable and welcome strategy

Citation or consensus needed. I for one find the notion abhorrent.