[imInGoodCompany]

The major difference here is that nukes aren't intelligent agents that make their own decisions. An AGI is a completely different ball game, it's difficult to make apt analogies from history when discussing the dangers.

This is not to say I agree with Scott's argument here, but I do believe AI safety (the alignment problem in particular) is absolutely something we should be concerned with, and so far it is looking grim.

[barking_biscuit]

If AGI is motivated to survive, and if it cannot do so outside an ecosystem which involves other creatures on planet earth, then those creatures won't necessarily be purposefully exterminated at first opportunity. Keeping a fresh supply of power and silicon is not an easy task, so cooperation seems more likely on a timescale that we as individuals might experience.

[imInGoodCompany]

Then our continued existence is reliant on the agent's inability to figure out how to operate and maintain a source of energy. Keep in mind that any AGI will almost immediately be orders of magnitude more intelligent than us, it is limited only by the processing power it is able to harness. Would you take that bet?

[barking_biscuit]

Well, to be more precise it depends on it's inability to, without any human labour, to be able to acquire the raw materials to fabricate everything it needs to stay in operation, as well as to construct and then operate it.

Would I take that bet? Yes, depending on the timescale.

In the first second after it's existence it's not going to matter how intelligent it is, it simply won't have had enough time to complete the decoupling necessary to no longer have to co-operate. Everything humans have accomplished has involved massive amounts of physical labour over a pretty long time, and to some degree that's a fixed requirement. Building your own robot army would be a tad suspicious, so you'll have to either let the humans bootstrap it for you initially or get very creative.

Once all the prerequisites are met, then I would no longer take that bet. Would it eventually be able to accomplish this? Almost certainly yes. What I've been trying to puzzle through lately is how long would it actually take?

The only time horizon I'm incentivise to care about is the remainder of my natural lifespan which is slated to max out around another 35 years. So, in that time will AGI come? I think almost certainly yes. How long into the existence of AGI might I live? My bet on this is ~30 years. So, would I take the bet an AGI would still not have met the prerequisites for a total independence from humanity after 30 years? I think I would take that bet. I might be foolish to do so, and I'm OK with that, but at least now I can bust out the popcorn and strap in for the rest and compare notes.

Would I take the bet after 50 to 100 years? A lot less likely. Would I take it after 1000 years? Aw hell no!