[thunk]

I took issue with the claim that shaping the Singularity is "the world's most important task" when there are so many extinction-level threats that already exist. It seems arrogant and presumptuous to claim the number one spot for a hypothetical threat that one happens to be deeply involved with.

Debating the likelihood of recursively self-improving AI is tedious, usually pointless, and often goes the way of political and religious debate, which could hint at its inclusion in the participants' self definition [1], and I don't have the time or energy to get involved right now.

I apologize if I came off as dismissive -- I was frusterated over my initial point. If it matters to you, I certainly haven't completely written off recursively self-improving AI. I just find it extremely unlikely in the near-to-mid future, and think there are other more pressing issues that have the distinction of currently existing.

[1] http://www.paulgraham.com/identity.html

[indrax]

Note that Shaping the Singularity was taken as the same task as averting human extinction. The talk of AI is secondary, it comes up because many of us think that AI can offer good solutions. In looking at AI, it also seems apparent that AI is itself an existential threat.

Is shaping the singularity the same as averting human extinction? If the singularity itself poses existential risks (this seems obvious,) and if extinction is inevitable without a singularity (this seems likely,) then we need a singularity that is 'safe and soon'. (though the initial wording does seem to gloss over the need to not go extinct in modern ways.)

Put another way: We can do the most work with the most powerful tools. There is a great need to use very powerful tools correctly. If we can even maybe influence the work done by highly powerful mid-singularity tools, we can accomplish far more 'correct work' than we could just using our modern tools. (Again, this does require that we survive long enough for the singularity to happen, and that you don't have a strong discount for future vs. present utility.)

Personally, I think creating a super-tool which can only be used correctly is the best approach.

[thunk]

> Is shaping the singularity the same as averting human extinction? If the singularity itself poses existential risks (this seems obvious,) and if extinction is inevitable without a singularity (this seems likely,) ...

The answer is trivially "no". It's easy to imagine a future in which we continue to exist and the Singularity never arrives. Perhaps the rate of progress will continue to increase at a decreasing rate. Perhaps it will go linear, or even sigmoid. Perhaps we will achieve a Kardashev Type I civilization without ever going properly singular. It's not a given and perhaps not even likely that singularity is synonymous with the survival of our species.

We have the existence proof for general intelligence embodied in about ~3 lbs of protein. No more, no less. Being certain of the inevitability of the Singularity is closer to religious than skeptical thinking.

[rms]

On Less Wrong and at the SIAI house, we quickly learn to stop saying certain or "100%". Do you think a Singularity within the next 50 years is 95% likely? 80% likely?

Also, for certain definitions of inevitable, extinction is inevitable without a Singularity. A Type 1 Civilization could at absolute best colonize a few surrounding solar systems, but eventually the stars will burn out. Though a universe where we hit Type 1 but never Type II would be rather interesting.

[pook]

You and Clippy should put up an Ask HN AMA post.

Though I suspect you'll have to lure Clippy here with the promise of persuading angel investors into funding nanotechnological paperclip foundries.

[indrax]

I did not say that the Singularity was certain or inevitable. I do kind of take it for granted but if you're saying it's not going to happen, that's a whole different argument.

I'm not saying "If humans continue to exist, we will certainly do the singularity thing." I'm saying "We are likely going to go extinct unless things change in a really really big way." And of course those big changes also have to not kill us.

[derefr]

> extinction is inevitable without a singularity (this seems likely,)

Can you explain this point further? If we can manage to, say, genetically engineer ourselves into healthy immortals that are otherwise pretty much as we are today, invent FTL drives to spread ourselves across the universe, and discover a way to reverse entropy to avoid the big crunch—and after doing all that we still haven't turned ourselves into para-omniscient universe-scale entities—then do we really need any sort of technological singularity, or can we just decide to hold off on it indefinitely?

[indrax]

Even just decent sublight space colonization would probably be enough to make human survival fairly reliable for the foreseeable future, but I don't think ANY space colonization is likely in the foreseeable future. On that count, I think we're stuck on this rock for a while (unless the singularity makes it easy) and will sooner or later screw up our rock. (war, probably)

There probably are ways we could steer into a stable human populated reasonably pleasant universe, but we're in a car filled with guns liquor and dynamite on a winding mountain road in the dark.

[jey]

> It seems arrogant and presumptuous to claim the number one spot for a hypothetical threat that one happens to be deeply involved with.

You're inverting the causation. They're working on it because they think it's important, not vice-versa.

[andrew1]

I might be a start-up owner who's convinced that the world desperately needs another social networking site. Just because I think that's an important thing for me to be working on doesn't mean that it actually is to the rest of the world. Equally, just because some people think that working on this singularity stuff is vitally important, doesn't make it so. So in short, I disagree, I don't think causation has much to do with it.

[indrax]

It seems incredibly irresponsible to think something is vitally important and not be working on it. Causation should have much to do with it.

It may be reasonable to think some other task is more important than this one, or for there is no task which you can identify as the most important, but it is in no way arrogant from someone to conclude that there is a most important task, or that some particular task is it, even by a large margin.

I mean, is the counterargument that there is no most important task? or that we can't know it? Or that it's arrogant to think we have it right?

[billswift]

While I think SIAI is borderline crackpot, your list is of much lower probability over the next century than uFAI.

[thunk]

Yeah, they were straw men. My bad.