[ilaksh]

I feel a medium-term or maybe even short-term threat (in the next 10-30 years) not of extinction necessarily but of becoming completely irrelevant. People like to jump to the conclusion that artificial super-intelligence will want to eliminate humans. I don't think that is a foregone conclusion at all.

However, if (when) super-intelligent artificial general intelligence "arrives", that pretty much makes normal unaugmented humans the relative equivalent of chimps. It means that our opinions and actions are no longer historically relevant. We will be, relatively speaking, obsolete mentally disabled people running along doing relatively stupid things. http://www.youtube.com/watch?v=I_Juh7Xh_70

In order for our opinions and abilities to actually matter relative to the super-doings and super-thoughts of the new AIs, we really _must_ have this magical nano-dust or something that integrates our existing homo sapien 1.0 brains with some type of artificial super-intelligence.

So that is what I am worried about -- will the super-AIs show up before the high bandwidth nano-BCIs (brain-computer interfaces) or before I can afford them.

Of course, in the long run there may not be a good reason for AIs to use regular human bodies/brains at all and so that may be phased out for subsequent generations.

[salmonellaeater]

Even an AI with innocuous goals will destroy humanity[1]. It would have to have explicitly pro-human goals in order to be anywhere near 'safe.'

[1] http://wiki.lesswrong.com/wiki/Paperclip_maximizer

[kamaal]

Well actually, no!

There are a lot of things to this Armageddon-through-AI equation. Firstly this thing called 'energy'. A paper clip maximizer will long reach its demise before converting a maximum part of planet to paper clips, merely because it needs a lot of resources to produce energy, and maintain itself too.

In fact true AI would not destroy anything. Because true AI would know it doesn't have sufficient data to make any such decisions.

[Houshalter]

There is more than enough energy on Earth alone to destroy humanity. Most likely an AI would want to maximize it's energy. Covering the entire face of the Earth with solar panels, or even building a matrioshka brain from the mass of the solar system.

An AI not specifically programmed to value human life would have no reason to keep us around. It'd either kill us because it sees us as a threat, competition, or just annoyance, or it would kill us by accident as it consumes all the resources it can.