[goolulusaurs]

Throughout history there have been hundreds, if not thousands of examples of people and groups who thought the end of the world was imminent. So far, 100% of those people have been wrong. The prior should be that the people who believe in AI doomsday scenarios are wrong also, unless and until there is very strong evidence to the contrary. Vague theoretical arguments are not sufficient, as there are many organizations throughout history who have made similar vague theoretical arguments that the world would end and they were all wrong.

https://en.wikipedia.org/wiki/Category:Apocalyptic_groups

[haswell]

If you were to apply this argument to the development of weapons, it’s clear that there is a threshold that is eventually reached that fundamentally alters the stakes. A point past which all prior assumptions about risk no longer apply.

It also seems very problematic to conclude anything meaningful about AI when realizing that a significant number of those examples are doomsday cults, the very definition of extremist positions.

I get far more concerned when serious people take these concerns seriously, and it’s telling that AI experts are at the forefront of raising these alarms.

And for what it’s worth, the world as many of those groups knew it has in fact ended. It’s just been replaced with what we see before us today. And for all of the technological advancement that didn’t end the world, the state of societies and political systems should be worrisome enough to make us pause and ask just how “ok” things really are.

I’m not an AI doomer, but also think we need to take these concerns seriously. We didn’t take the development of social networks seriously (and continue to fail to do so even with what we now know), and we’re arguably all worse off for it.

[hackermatic]

Although I think the existential risk of AI isn't a priority yet, this reminds me of a quote I heard for the first time yesterday night, from a draft script for 2001: A Space Odyssey[0]:

> There had been no deliberate or accidental use of nuclear weapons since World War II and some people felt secure in this knowledge. But to others, the situation seemed comparable to an airline with a perfect safety record; it showed admirable care and skill but no one expected it to last forever.

[0] https://movies.stackexchange.com/a/119598

[adverbly]

Fun! Let me try one:

Throughout history there have been millions, if not billions of examples of lifeforms. So far, 100% of those which are as intelligent as humans have dominated the planet. The prior should be that the people who believe AI will come to dominate the planet are right, unless and until there is very strong evidence to the contrary.

Or... those are both wrong because they're both massive oversimplifications! The reality is we don't have a clue what will happen so we need to prepare for both eventualities, which is exactly what this statement on AI risk is intended to push.

[goolulusaurs]

> So far, 100% of those which are as intelligent as humans have dominated the planet.

This is a much more subjective claim than whether or not the world has ended. By count and biomass there are far more insects and bacteria than there are humans. It's a false equivalence, and you are trying to make my argument look wrong by comparing it to an incorrect argument that is superficially similar.

[cwkoss]

No individual human has ever dominated the planet

[mitthrowaway2]

Many people seem to believe that the world is dangerous, and there are things like car accidents, illnesses, or homicides, which might somehow kill them. And yet, all of these people with such worries today have never been killed, not even once! How could they believe that anything fatal could ever happen to them?

Perhaps because they have read stories of such things happening to other people, and with a little reasoning, maybe the similarities between our circumstances and their circumstances are enough to seem worrying, that maybe we could end up in their shoes if we aren't careful.

The human species has never gone extinct, not even once! How could anyone ever believe that it would? And yet, it has happened to many other species...

[_a_a_a_]

> So far, 100% of those people have been wrong

so far.

[jackbrookes]

Of course every one has been wrong. If they were right, you wouldn't be here talking about it. It shouldn't be surprising that everyone has been wrong before

[goolulusaurs]

Consider two different scenarios:

1) Throughout history many people have predicted the world would soon end, and the world did not in fact end.

2) Throughout history no one predicted the world would soon end, and the world did not in fact end.

The fact that the real world is aligned with scenario 1 is more an indication that there exists a pervasive human cognitive bias to think that the world is going to end, which occasionally manifests itself in the right circumstances (apocalypticism).

[staunton]

That argument is still invalid because in scenario 2 we would not be having this discussion. No conclusions can be drawn from such past discourse about the likelihood of definite and complete extinction.

Not that, I hope, anyone expected a strong argument to be had there. It seems reasonably certain to me that humanity will go extinct one way or another eventually. That is also not a good argument in this situation.

[goolulusaurs]

It depends on what you mean by "this discussion", but I don't think that follows.

If for example, we were in scenario 2 and it was still the case that a large number of people thought AI doomsday was a serious risk, then that would be a much stronger argument for taking the idea of AI doomsday seriously. If on the other hand we are in scenario 1, where there is a long history of people falling prey to apocalypticism, then that means any new doomsday claims are also more likely to be a result of apocalypticism.

I agree that is is likely that humans will go extinct eventually, but I am talking specifically about AI doomsday in this discussion.

[haswell]

> If on the other hand we are in scenario 1, where there is a long history of people falling prey to apocalypticism, then that means any new doomsday claims are also more likely to be a result of apocalypticism.

If you're blindly evaluating the likelihood of any random claim without context, sure.

But like the boy who cried wolf, there is a potential scenario where the likelihood that it's not true has no bearing on what actually happens.

Arguably, claims about doomsday made now by highly educated people are more interesting than claims made 100/1000/10000 years ago. Over time, the growing collective knowledge of humanity increases and with it, the plausibility of those claims because of our increasing ability to accurately predict outcomes based on our models of the world.

e.g. after the introduction of nuclear weapons, a claim about the potentially apocalyptic impact of war is far more plausible than it would have been prior.

Similarly, we can now estimate the risk of passing comets/asteroids, and if we identify one that's on a collision course, we know that our technology makes it worth taking that risk more seriously than someone making a prediction in an era before we could possible know such things.

[jabradoodle]

What constitutes strong evidence? The obvious counter to your point is that an intelligence explosion would leave you with no time to react.

[goolulusaurs]

Well, for example I believe that nukes represent an existential risk, because they have already been used to kill thousands of people in a short period of time. What you are saying doesn't really counter my point at all though, it is another vague theoretical argument.

[jabradoodle]

It was clear that nukes were a risk before they were used; that is why there was a race to create them.

I am not in the camp that is especially worried about the existential threat of AI, however, if AGI is to become a thing, what does the moment look like where we can see it is coming and still have time to respond?

[goolulusaurs]

>It was clear that nukes were a risk before they were used; that is why there was a race to create them.

Yes, because there were other kinds of bombs before then that could already kill many people, just at a smaller scale. There was a lot of evidence that bombs could kill people, so the idea that a more powerful bomb could kill even more people was pretty well justified.

>if AGI is to become a thing, what does the moment look like where we can see it is coming and still have time to respond?

I think this implicitly assumes that if AGI comes into existence we will have to have some kind of response in order to prevent it killing everyone, which is exactly the point I am saying in my original argument isn't justified.

Personally I believe that GPT-4, and even GPT-3, are non-superintelligent AGI already, and as far as I know they haven't killed anyone at all.

[usaar333]

> Personally I believe that GPT-4, and even GPT-3, are non-superintelligent AGI already, and as far as I know they haven't killed anyone at all.

They aren't agentic. There's little worry a non-agentic AI can kill people.

Agentic AI that controls systems obviously can kill people today.