[charlesdenault]

If we create machines that learn as well as our brains do, it’s easy to imagine them inheriting human-like qualities—and flaws. But a “Terminator”-style scenario is, in my view, immensely improbable. It would require a discrete, malevolent entity to specifically hard-wire malicious intent into intelligent machines, and no organization, let alone a single group or a person, will achieve human-level AI alone.

Isn't this incredibly shortsighted? Ignoring all the questions regarding the morals and ethics an intelligent machine may feel and affect the way it behaves... It used to take nations to build computers, then large corporations, then off-the-shelf parts by a kid in his garage.

The first strong AI will most likely be a multi-billion dollar project, but its creation will arguably usher in an era in which strong AI is ubiquitous.

[emiliobumachar]

No malice is necessary.

https://wiki.lesswrong.com/wiki/Paperclip_maximizer

"The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."

[wapapaloobop]

The idea that I'll use human bodies if I need a bunch of atoms makes as much sense as the premise of The Matrix i.e. harvesting humans for energy production.

[JumpCrisscross]

I don't think the line is meant to be taken literally. Consider the myriad of environmental factors humans need to remain alive. Any of those factors, e.g. the atmosphere's current makeup, could prove uninteresting to an AI.

[gruturo]

I read that the first draft was using a much more reasonable concept - the Matrix needed our brains as CPUs to run itself. No idea if it's true, but certainly a lot less implausible than what came out in the movie.

[qbrass]

The power source thing is an instance of unreliable narration. Their account of what happened is pieced together and passed around like the mythology and religion that underscores the movie.

[charlesdenault]

Humans need a bunch of atoms (organic molecules) to survive, and we source that from other living organisms. That parallel isn't too far off.

[wapapaloobop]

Humans need a bunch of atoms (stone, minerals) to survive, and we source those from quarries. This parallel is closer.

[kaffeinecoma]

I think the concern is more that a sentience that is many orders of magnitude more intelligent than humans simply won't care about us. How do you feel about ants?

[fulafel]

Humans use other mammals for all kinds of stuff and AI will be seeded with our knowledge. Humans are plentiful: https://xkcd.com/1338/

[wapapaloobop]

Any person (biological or artificial) who is capable of mass murder is unlikely to balk at slave owning. If he really wanted human-sized parcels of matter he'd be better off putting people to work in mines. A human can collect many times his own mass per day.

[squeaky-clean]

> If he really wanted human-sized parcels of matter he'd be better off putting people to work in mines.

But the AGI in this thought experiment doesn't want human-sized parcels of matter. It wants paperclips. And by proxy, it wants any matter which is not a paperclip and can be transformed into a paper clip.

> A human can collect many times his own mass per day.

And if the AI can collect many thousands of humans worth of mass each minute, it doesn't care about the humans or their ability collect things. The AGI simply wants to optimize the number of paperclips in it's collection, it doesn't care about number of humans working for it, or that it has displaced or used to make paperclips, unless they affect how many paperclips it can acquire.

I like the ants metaphor another commenter made and want to continue with it. When digging a pit / quarry, or clearing land or whatever.... Well, ants can lift several times their bodyweight and dig some deep holes. But do we use the ants to dig for us? Or do we bring in a backhoe and not care whether ants exist in the field?

Also, your scenario still agrees with the thought experiment. It's not that an AI will intentionally kill all humans. Just that a super-intelligent AI designed without consideration for human values will not likely develop those values on it's own, and therefore can be just as dangerous as an AI designed to be malicious towards those values. In your example, a machine AI has still enslaved the entire human race, and it's original instructions were not at all malicious or related to destroying humanity.

[Eliezer]

This is an excellent analogy! "Ants can carry fifty times their own weight. Why would humans wipe out the ants on a construction site instead of using food rewards to train them to help remove dirt?" Answer: backhoes are faster at moving dirt, the time lost by carefully moving the ant colony exceeds any gain from their helping to move the dirt on one tiny section of the project, ants are hard to trade with and you wouldn't be sure they were doing the job right, and humans don't have a terminal value for protecting the ants.

And that's without a button that disassembles the ants and uses their atoms to build part of a diamondoid backhoe that's much better at moving dirt.

[wapapaloobop]

The problem with your scenario is that it equates humans with animals, but the difference that matters is between people (humans, AGIs) and non-people (primitive replicators, animals).

We don't commit murder to get at rocks despite the fact that weren't designed with 'consideration for human values'. We learn our values, just as an AGI would, not through programming or lack thereof, but through parenting, education and culture. These latter things aren't optional since minds can't exist without them.

[fulafel]

People use dead mammals for things like leather, bones, fat, feeding other animals/humans, science experiments, ornaments/status symbols, tallow, etc which you can't mine. The AI might be in a hurry (eg be in competition or war with humans other AIs) and trade off long term cooperation prospects for getting an upper hand in a fast paced emergent situation.

I'm not sure getting sorted into two piles, beasts of burden and recycled biomass, is really a win scenario for humans either.

[Zigurd]

If you consider the extent to which humans have shaped the Earth's environment to suit them, we massively favor animals we kill and eat over the ones we don't. You don't have to be part of the machine to be part of a congenial environment for the machine.

[vkou]

We already live in this world. Capitalism is by design, a paperclip maximizer[1].

Whether or not this means we live in a hellish dystopia is in the eye of the beholder.

[1] https://thoughtinfection.com/2014/04/19/capitalism-is-a-pape...

[Eliezer]

Nice derail! But the fact that there isn't yet a 400-lightyear expanding hole in the Milky Way suggests that corporations are not, in fact, as dangerous as unaligned machine superintelligences are being alleged to be.

[vkou]

The fact that there isn't yet a 400-lightyear expanding hole in the Milky Way suggests that corporations have not yet figured out how to be as dangerous as science-fiction mechanical superintelligences are alleged to be in their quest to maximize paperclips.

If we don't think that AI can be safely controlled, why do we think that corporations can be? Likewise, if we think that they can, why not the converse? Which fail-safes are effective for one, but not the other?

[conceit]

> why not the converse

In that regard, the term superhuman-AI is used.

[conceit]

Capitalism is rather a human worth maximizer. It's not a money maximizer.

[fratlas]

I believe this is the most likely scenario: lethal apathy. I mean, how could a turing machine have emotions without chemicals?

[drabiega]

Emotions are a heuristic. Any general AI powerful enough to be useful is going to have to be full of similar heuristics if it's going to get anything done. The presence of chemicals is irrelevant.

[maroonblazer]

In what way are emotions a dependency on getting anything done? The AI could be like the high-functioning sociopath who doesn't actually experience the subjective feeling of emotion but has learned all the physical cues that convey a given emotion. An AI would execute them flawlessly.

[adrianN]

What have emotions to do with this? And what have chemicals to do with emotions?

[fennecfoxen]

The Terminator scenario is basically one, single centralized AI wired into everything (or at least wired into enough military power to stave off opposition as it force-wires itself into all the rest of the things). But practically speaking, a single centralized AI is highly unlikely to be a world-takeover machine of godlike intelligence in its first iteration. The iterative improvement that you refer to which could potentially make it dangerous also points to a likelihood of there being a diversity of agents with a variety of designs being run by a diverse set of organizations with a diverse set of goals, with many incentives to obtain protection from hostile AIs and computer systems run by 1337 h4x0rz (possibly foreign government hackers).

Diversity and independence of AI Agents mitigates against the Terminator-scenario danger; the situation would really be not so different than the modern-day situation with natural intelligences that have control over resources and weapons of war.

[Houshalter]

>Diversity and independence of AI Agents mitigates against the Terminator-scenario danger

I don't see how this is the case. The AI's still are likely to be more similar to each other than they are to humans. And they could choose to cooperate to kill humans, and then split the resources of the world among themselves.

Having multiple AIs is no guarantee of safety! Unless those AIs want to keep humans alive and care about our future.

The second assumption is that the world of the future will be stable. It could be that a small technological advantage is enough to win against every other group. E.g. if your AI can invent better nanotechnology first, or discovers some entirely unknown technology. It's unlikely that a random set of superintelligences would be perfectly evenly matched. Some would be much larger, smarter, or have access to more resources, or have a head start in time.

Lastly and most importantly, is the idea that superintelligence won't be built quickly. Once you have the first smarter than human AIs, it may be only a matter of weeks or days before you have superintelligence. That's the whole idea of an "intelligence explosion". The smarter than human AI's can do AI research and improve themselves, and then improve themselves even more, then even more. Not to mention plugging more GPUs into their brains to make them smarter.

So it seems very likely to me that once you have a lab that solves the last technical problems in creating AGI, a single superintelligence will rapidly emerge from that. The superintelligence will then be able to get whatever it wants, because it's so much smarter than humans. It doesn't need to be "wired into" anything if it's 10x smarter and faster than the best human hackers. And if it's 2050 and everything and your toothbrush is connected to the internet.

[pka]

I'm not qualified enough to comment on the possibility of strong AI, but one thing to consider is that natural intelligences don't launch the missiles only because they are afraid of the resulting total annihilation, of death basically. Maybe globally distributed strong AIs won't have such concerns.

[gm-conspiracy]

I am not so worried about explicit physical violence, but subtle manipulation of what us humans consider reality.

Also, are you assuming the iterative development requires human intervention?

[TheOtherHobbes]

If anyone is really worried about magic AI, they can build it inside a VR bottle and add failsafes to protect against god-like supersentience.

I think god-like supersentience is ludicrously unlikely, if only because it would probably have to be NP complete and bug free, and both seem like a very tall order - you immediately run into the barber's paradox where you're expecting a system to understand itself completely using its own system.

Is this likely? I'd say no. In fact I conjecture it's physically and metaphysically impossible.

More mundane super-AIs are possible, but they reduce to the out of control machine problem.If there's a problem, it will be because of the possible destructiveness of the implementation technology - e.g. nano, or even traditional bio - and not inherently because of AI.

Bio-like systems may turn out to be more efficient than silicon, so we're far more likely to have a problem if we build a smart machine self-evolving machine out of DNA than out of GPUs.

Arguably we're a smart self-evolving machine already, and the jury is still out on whether or not we're a good idea.

[Houshalter]

>In fact I conjecture it's physically and metaphysically impossible.

That's crazy. Human brains were created by nothing more than random mutations and selection. A very stupid algorithm created the most intelligent thing that currently exists. Humans are a thousand times smarter than evolution. We can invent things evolution never could.

And now humans are learning about how our own brains work through studying them, and also how to build intelligent algorithms ourselves. And we are doing it much faster than evolution did (10's of years vs millions.)

>the barber's paradox where you're expecting a system to understand itself completely using its own system.

I don't see how this is a paradox. There's no reason a system can't understand itself. The algorithms in your brain are probably much simpler than the amount of space your brain has to store information.

>Bio-like systems may turn out to be more efficient than silicon

Silicon is already vastly more efficient. Transistors can be the width of atoms, while synapses are the widths of many molecules. Neurons have to maintain all the machinery necessary for life and self replication. They use slow and inefficient chemical signals and reactions. Etc, etc.

[JumpCrisscross]

> If anyone is really worried about magic AI, they can build it inside a VR bottle and add failsafes to protect against god-like supersentience

This is sort of like a dog trying to trap its human while it raids the pantry. You have to assume, whether through technical superiority or deception, the AI will figure out how to escape (re-read Lord of the Rings imagining the One Ring as an AI).

[Houshalter]

It also assumes that humans would want to trap it. If you wanted to do that, why would you build it in the first place? The people that build the first AIs will be those like the parent commentator, that don't take the risk seriously.

[gm-conspiracy]

My concern is that the builders and the worriers are not the same entity.

[rjsw]

What if the AI just thinks that it understands itself ?

[fennecfoxen]

> Also, are you assuming the iterative development requires human intervention?

At first, yes; long-term, it doesn't matter too much. If IBM's AI agents come up with better versions of themselves they'll still be trying to rent them out to different people in order to compete with Google's and Baidu's and Lockheed's and Siemens' businesses in those market (or whoever's still around and actually running these things at that point).

[gm-conspiracy]

So, multiple benevolent feral AIs that coexist in harmony?

[snemvalts]

Andrew Ng made a really good analogy to those afraid of strong AI destroying humanity: "It's like being afraid of overpopulation on Mars, we haven't even landed on the planet yet."

[FeepingCreature]

To be fair, we do worry about contamination of Mars with microorganisms, which I believe is a better analogue for something with a potential exponential takeoff.

[ehudla]

It's more like we haven't discovered Mars yet.

[Houshalter]

It's a stupid analogy. Mars overpopulation would obviously take many, many centuries. It would be a slow thing that you could obviously see coming. There is no reason to believe AI will take centuries to build, or that we will necessarily see it coming.

A better example might be like H.G. Well's 1913 prediction of nuclear weapons destroying the world. It was something that science was just realizing was possible, and would be invented within his lifetime.

[snemvalts]

We're far from emulating networks on the scale of the visual cortex, let alone a self-reasoning machine (we don't even fully understand consciousness and inner workings of the brain).

People fearing strong-AI are the ones not involved in the field, yet all this hype/fear from them (in combination with Moore's law ending) is probably going to cause another AI winter.

[Houshalter]

And in 1913 we didn't have even basic nuclear technology. Just 3 decades is a long time for newly emerging technologies.

>We're far from emulating networks on the scale of the visual cortex

In 2009 (ish) computer vision was a joke that could recognize very few objects a small percent of the time. Based on only simple color and texture, and sometimes basic shapes.

A few years later and computers were excelling at computer vision recognizing a majority of objects. A year or two after that, and they started to beat humans on those tasks. We already have super-human visual cortexes. Who knows what will be possible in a decade.

We will probably never understand the inner workings of the brain. Not because it's complicated, just because reverse engineering microscopic systems is really hard (imagine trying to reverse engineer a modern CPU vs merely designing one.) Especially hard because we can't ethically dissect living humans and do the experiments we would need to do.

But that's no concern, AI advances on from first principles. AI researchers invent better and better algorithms every day, without having a clue what neuroscientists are up to.

>People fearing strong-AI are the ones not involved in the field,

That's just incorrect. A survey of AI researchers found they give about a third chance AI will turn out badly for humanity in the next century: http://www.nickbostrom.com/papers/survey.pdf

>We thus designed a brief questionnaire and distributed it to four groups of experts in 2012/2013. The median estimate of respondents was for a one in two chance that highlevel machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be ‘bad’ or ‘extremely bad’ for humanity.

>in combination with Moore's law ending

Computers can advanced a long time after Moore's law. Google just released a special neural network chip that is equivalent to 7 years worth Moore's law. 3d architectures can vastly increase the number of transistors. Better algorithms can make NN's that require many fewer transistors to do computations, or even do cheap analog computations.

[sedachv]

> In 2009 (ish) computer vision was a joke that could recognize very few objects a small percent of the time. Based on only simple color and texture, and sometimes basic shapes.

This is completely inaccurate and totally ignores the history of machine vision.

Computer vision was in no way a "joke" in 2009. OCR and manufacturing inspection systems have been successfully deployed since the 1980s. Neural networks were being applied to computer vision in autonomous vehicles in 1989: https://www.youtube.com/watch?v=ilP4aPDTBPE

> We already have super-human visual cortexes.

No we don't: http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.ht... (see also the HN discussion: https://news.ycombinator.com/item?id=9749660)

I remember reading about a similar thing that happened in the 1980s to some DARPA funded project that was trying to apply neural networks to tank/vehicle detection: the network got really good at recognizing the foliage that the training images had in them.

Robust scene understanding is a very hard problem and still far from solved. Again, research on this has been going on since the 1960s.

> But that's no concern, AI advances on from first principles. AI researchers invent better and better algorithms every day, without having a clue what neuroscientists are up to.

Do you realize what the 'neural' in neural networks refers to? People working on AI did not suddenly stop paying attention to neuroscience after Perceptrons were invented.

[conceit]

> imagine trying to reverse engineer a modern CPU vs merely designing one

I can't really imagine the latter to begin with.

[xyience]

Sometime in the 2020s, Elon Musk sends the first team of 5 astronauts to Mars... guess what, Mars is now overpopulated. Ng might not want to worry about it but be thankful other people are, lives are on the line.

[hasbroslasher]

It's more like "We're killing our own planet via overpopulation but are more prone to argue about theory on the internet"

[Unklejoe]

I think a lot of the fear comes from the fact that if/when such a system is created, its knowledge and capabilities will only be equal to a human's for a short period of time, after which it will probably surpass any human capability at an ever increasing rate. Then, we would be suddenly at the mercy of it, which scares a lot of people. It’s very hard for us to try and predict the actions of something that could end up being thousands of times smarter than us.

[mtgx]

Yeah, did we forget the DAO "hack" just happened?

AIs can be manipulated, too, even when they "work as intended." People are putting too much trust in AI because it's "just math" and "just an objective machine". Maybe it won't make the same errors as humans would, but it could make a whole lot of other types of "errors" (at least from a human perspective). And who's to say the deep neural net algorithms written by humans aren't flawed to begin with?

[outworlder]

> Isn't this incredibly shortsighted?

Yes, it is.

> no organization, let alone a single group or a person, will achieve human-level AI alone

This is completely irrelevant. Done once, it can be replicated. Even worse, if we reach human-level AI, it can be done by the AIs themselves.

It is also anthropomorphising machines. They don't need to be malevolent. They just need not to care, which is much easier.

Should an ultra-intelligent machine decide to convert New York into a big solar power facility, it wouldn't necessarily care to move the humans out first.

[nkozyra]

I get the impression most experts in the field are trying to downplay this. Not because it's impossible, but because it hurts the image of AI.

As long as people are aware of the real possibility of AI working against human interests or ethics, we can ensure there are safeguards asking the way.

But a hand wave "won't happen" to the general public is assuredly a PR move.

[saiya-jin]

it would be perfectly OK if we would be living in peaceful world without any conflict, without nations which should hold the banner of democracy and freedom, where there are no terrorist attacks and so on.

last time i checked, we're getting further and further away from this scenario. in fact, we SHOULD expect evil AI in worst form possible and beyond, anything else is dangerously naive.

[fennecfoxen]

We should expect evil AI. But it's not patently obvious that we should expect it to make the world worse off than evil natural intelligences. The horrors of World War II and the Holocaust were accomplished using humans, and global thermonuclear annihilation was on the table while computers were still programmed with core rope memory strung together by little old ladies with knitting skills.

Perhaps there is something specific about the availability of artificial intelligences, and their relation to the world around them, that is different and intrinsically more threatening than a world containing only human intelligences - but surely we mustn't leap to that conclusion without identifying what this difference is.

[ZanyProgrammer]

"We should expect evil AI."-with your certainty, I'd appreciate some lottery numbers.

[fennecfoxen]

Honestly, if nothing else, someone will make it their pet project as a sick joke. Of course, that doesn't mean there'll necessarily be a high-quality or impactful evil AI, mind you... but barring global thermonuclear annihilation it will happen sooner or later.

Your lottery numbers are 23skidoo. Your I Ching reading is ䷍ (Wealth, great treasures, supreme success).