[ericjang]

I'm with you on this, and I logged in just to upvote your comment.

That said, if one suspects that humanity will not be able to construct powerful computer systems in a provably friendly manner ("friendly AI"), shouldn't we step back and question this Promethean effort?

I know this is what AI researchers caution against - hysteria stunting basic science research, AI winters, etc. However, if someone speculates that any progress in AI research leads us down a path that, in the long term you speak of, will eventually cause humanity to suffer, then shouldn't he or she do everything they can to stop it?

Of course, whether or not it is feasible to stop technological progress is a separate question.

[seiji]

I'm with you on this, and I logged in just to upvote your comment.

Thanks! :)

shouldn't we step back and question this Promethean effort? is feasible to stop technological progress is a separate question.

The only way to stop it would be to classify GPUs as weapons of mass destruction. But, that's a non-starter because all our fabs aren't US based, so if we restricted domestic GPU usage, then fer'ners would just use them all and create god forms in their own image.

Isn't it amusing how when people think "friendly AI," the "AI" always seems to have the motivations and personality of a 25 year old startup scene white guy living in San Francisco? What if the AI is made by a fanatic in a marginalized country? That's the "friendly-vs-not-friendly" dichotomy, but there's no way to force one way or the other (without restricting hardware access).

Over all, global capitalism doesn't really allow for holding anything back for the sake of "a better world." We stand on the broken backs of others so we, ourselves, can rise taller, thereby proving we are better than the unwashed masses. We are special individuals because we have the vision, audacity, power, to exploit the rest of the world en masse. Tremble before our bank accounts, mansions, and political connections.

then shouldn't he or she do everything they can to stop it?

Thought experiment: what if "what comes next" is better than humanity? Industrial-scale human civilization so far hasn't been great for other life on this planet.

What if "regular" apes had the forethought to kill the antecedent human mutated apes because it would eventually be bad for the planet? Would restricting advancement have been a better plan than letting new intelligence be birthed, even if it destroyed all the old ways?

The question ends up being: do we commence a Butlerian Jihad or do we admit we are imperfect meat machines and, perhaps, tens of billions of meat forms aren't ideal citizens of anything given a long enough time horizon?

[mojuba]

> Thought experiment: what if "what comes next" is better than humanity?

Better from whose perspective? :)

The machines can potentially be more durable, and also scalable. We are fragile, we don't scale very well (though trying hard through globalization). Today we can't even backup/restore a person's mind. In fact I think self-aware intelligent machines will appear sooner than the technology to backup our own biological brains.

They will scale their brains, they will expand their "natural habitat" towards other planets, solar systems and even galaxies, like those all-eating self-replicating automatons. Easily.

So while we are at it, looking at the grand scale of things, just one question: where are all these machine civilizations in the Universe? Why aren't there any traces of exponentially growing all-consuming machine civilizations anywhere?

[seiji]

Why aren't there any traces of exponentially growing all-consuming machine civilizations anywhere?

I'm a fan of the zoo hypothesis. Everything else is just sufficiently "hidden" from observation currently. I mean, if you had a planet full of humans, would you want to talk to them?

The zoo hypothesis seems to breaks down into two underlying reasons for non-contact though: captivity vs. captivating. Either it's a star-trek like "non-interference" thing keeping us isolated for future self-directed development/destruction — or — we are just so frigging boring/hostile/dumb it isn't even interesting to consider engaging us.

(those two reasons do assume intent and purpose on the all-consuming machine civilizations, which we have zero priors for. are there clusters of grey goo out there mindlessly absorbing galaxies as they travel? are there megamind clusters trying to do... something? is this all just a simulation? will the real slim shady please stand up?)

[rl3]

>Would restricting advancement have been a better plan than letting new intelligence be birthed, even if it destroyed all the old ways?

In other words, ban technology or give up. Of course, banning technology is fruitless for the reasons you outlined.

Arguably the only sane solution is to attempt building AGI in the safest and most ethical way possible, and doing it before anyone far more reckless or unethical can.

Granted it'll still probably fail and destroy us all (controlling something smarter than you is really hard), but putting forth humanity's best effort sure beats surrendering to fate—or worse, the aforementioned fanatic in a marginalized country.

[AnimalMuppet]

You're presuming that there will only be a single origin. I think that's false. Even if "we" build AGI "in the safest and most ethical way possible", there will be a "they" who doesn't.

[rl3]

A multipolar scenario is possible but unlikely. If the technology easily scales, then whoever arrives at it first will probably win.

[AnimalMuppet]

If the technology creates a rapid positive feedback loop, singularity-style, then I might agree. If not, though... well, if not, it won't matter as much, because there will be less to fear from an AI that can't rapidly improve itself.

[XFrequentist]

The more precise version of "friendliness" is value alignment - Stuart Russell describes the problem well here[1], if you're interested.

1: http://edge.org/conversation/the-myth-of-ai#26015

[Chathamization]

Most fields have the potential to cause humanity to suffer if we look far enough into the future and look at what might be possible rather than what is likely. GMO research could possibly create superpredators that eat humans, neuroscience could possibly create superorganisms that consists of linking multiple human (or non-human) brains, a futuristic near-light speed spaceship could be devastating to the earth if it ever accidentally hit it, etc.

These only exist in the realm of Science Fiction at the moment, but the same is true for doomsday AI.