[froggychairs]

Not OP, but I do work in reducing harms of deployed AI systems at a big tech company

When most EA/Rationalist folks discuss AI Safety. They’re talking about AGI, a hypothetical construct that we have no way of proving is even possible

Now there’s nothing inherently wrong with funding research of that sort.

The problem comes is that many EA/Rationalist folks begin downplaying legitimate tangible risks on the horizon (I’ve had prominent EA folks tell me that Climate Change is not an existential risk) or that many of these AI Safety researchers actively ignore how to reduce harms of the AI we have deployed today.

This existential measuring contest is just so odd and unnecessary. I really don’t care if you think AI Safety is a legitimate problem. I personally don’t but it’s when they begin to downplay others actual catastrophes that many of us take issue.

[FeepingCreature]

The atom bomb was once a hypothetical construct. The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible. Proof is irrelevant outside of mathematics; the only relevant question is cost to expected benefit - or expected harm.

With dangerous research, when a conclusive demonstration of possibility is achieved, it is usually too late to do much in the way of prevention. That's why we don't expect the FDA to "prove" that harm from a new medication "is even possible" before regulating it; rather, we expect the pharmaceutical company to provide evidence for its expected harmlessness before we even make the attempt. Think of the FDA what you like, this is certainly a better standard than putting the burden of proof on harmful outcomes.

[danaris]

Part of the problem with this is that conclusive demonstration that AGI is possible is not even in the same ballpark as a conclusive demonstration that it would have consequences like the EA folks seem to believe.

From everything I've read about their views, they seem to believe that either the moment an AGI is created or very, very soon thereafter, it will achieve the Singularity and "ascend" to something near godhood from our perspective, and it will perceive humanity as a threat to it and immediately seek our destruction—which, because of its near-godhood, it will be able to achieve far faster than we have the ability to respond to. Thus, what we have to do is either prevent AGI from ever coming to pass (because as soon as it exists, it's effectively too late), or make absolutely certain that by the time it does, we have strong measures in place to either fight back against it, or get humanity the hell out of Dodge.

This is built on such a foundation of shaky assumptions that giving it credibility is ludicrous.

[FeepingCreature]

I agree with your description of the AGI/ASI position.

My take on it is the other way around: every step of the argument seems plainly obvious.

[yifanl]

Plainly, that is not the case, as you are choosing to not spend your every waking moment to stop the antichrist AI coming into existence.

So either it's less obvious than you're claiming, or you are taking the threat of creating an evil AGI not seriously enough according to your own beliefs.

[FeepingCreature]

I don't spend every waking moment stopping lots of things that could kill us all. Nuclear war, climate change, gain-of-function research.

I just don't care that much about things that kill everyone. I don't think almost anyone does. In the Cold War, the vast majority of the population was not doing absolutely everything to achieve bilateral disarmament. This would lead you to believe that they didn't really expect nuclear war. But maybe they were just apathetic and helpless.

[pvg]

The atom bomb was once a hypothetical construct. The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible.

This isn't an accurate representation, really. The letter was written exactly because it was clear that a fission bomb is very likely within the short-term reach of current technological and industrial capability. This is very much not the case with AGI.

[FeepingCreature]

That's a difference of degrees, not kind. You can always ask for a higher class of proof.

[pvg]

How is it a difference of degrees? This is just plain inaccurate:

The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible.

Supporting your argument with an inaccurate thing is not 'a difference of degrees'. It's a difference between a good argument and a bad one.

[FeepingCreature]

The disagreement is about the difficulty of "proving". I could say that unfriendly AI is clearly possible because evil humans exist. You could say that unfriendly AI is not proven possible until one has actually been built. But if you applied that standard to the letter, it would likewise fail it. Now, I agree that the case for AI is weaker, maybe much weaker, than the case for nuclear fission when the letter was written - there is, for instance, no clear research agenda that is held by preeminent researchers in the field to lead to it - though there are research groups, such as Deepmind, that have AGI as their explicit goal - but that also has nothing to do with being "proven" possible.

My argument is that for contested and dangerous outcomes, proof of possibility is an unreasonable standard, in good part because it massively underspecifies the goalposts.

[pvg]

The disagreement is about the difficulty of "proving"

It's not, it's just a turn of phrase you've latched on to. There's no discussion of a standard of proof, etc. The comparison between the Szilard Einstein letter and 'AGI safety' is specious and it's one you brought up! You have to outright misrepresent what the letter was about just to make it. It was not a letter about a 'hypothetical danger they had no way of proving'. There just isn't any reasonable reading of the letter, the context, the history in which that's an accurate statement.

[concordDance]

> I’ve had prominent EA folks tell me that Climate Change is not an existential risk

Have you seen anything to indicate otherwise? The IPCC reports don't contain even a whiff in that direction.

To the best of my knowledge zero people who know anything about climate change think its existential.

[andybak]

Is there an accepted unambiguous definition of "existential" here?

Is it:

1. All life on the planet dies

2. All advanced life on the planet dies?

3. All humans die?

4. Advanced civilization is destroyed irretrievably?

5. Advanced civilization is destroyed for a long period of time?

etc

And I'm not sure the group "people who know anything about climate change" are neccesarily better equipped to answer this question - it's a complex systems question.

I'm not actually sure who is equipped to answer it accurately.

[a3_nm]

It looks like there are research works trying to define precisely this notion of existential risk in the context of climate change, see e.g. https://link.springer.com/article/10.1007/s10584-022-03430-y

[danaris]

I'm not sure if this would actually count as "existential", but I think that it's well worth being very, very concerned about a consequence on the order of

"Millions to billions of people die; large percentages of the rest are displaced and/or have their lives made significantly shorter, harsher, and less certain as war, famine, disease, and natural disasters become vastly more common around the globe."

And there's really very little question that that's where climate change is leading if we don't get it under control soon.

[concordDance]

It's always 2 or 3 in my experience.

[edanm]

> The problem comes is that many EA/Rationalist folks begin downplaying legitimate tangible risks on the horizon (I’ve had prominent EA folks tell me that Climate Change is not an existential risk) or that many of these AI Safety researchers actively ignore how to reduce harms of the AI we have deployed today.

I'm sure some people do this, and it's bad.

But to be clear, when you say some researchers are actively ignoring how to reduce harms of AI today... I'm not sure what you mean by "actively" ignore, but don't most researchers just by default ignore this, except for the few who are active in this field?

In other words, I think it's totally legit to focus on long-term AGI risk, and totally ignore current short-term AI risk, as a personal career-choice, just like it's ok to not even work on AI at all. Unless someone is actively trying to cause less resources to go into short-term AI risk, what's the problem?

> They’re talking about AGI, a hypothetical construct that we have no way of proving is even possible

I mean, we exist as (one would hope) intelligent beings. Why wouldn't AGI be possible? Your frame makes it seem like the default is that it isn't possible, which is wrong IMO.

[in3d]

They are right. Climate change is absolutely not an existential risk.

“In a high-emissions scenario where little is done to curb planet-heating gases, global mortality rates will be raised by 73 deaths per 100,000 people by the end of the century”

https://www.theguardian.com/us-news/2020/aug/04/rising-globa...

[naasking]

> Climate change is absolutely not an existential risk.

That's an assessment based on what we know, but there's a lot we don't. For instance, it's possible that climate change could lead to a large scale collapse of food chains that could pose an existential risk to humanity.

It's also plausible that climate change will lead to resource shortages that result in wars between nuclear powers, and the nuclear calamity that follows kills us all.

[TheCoelacanth]

"By the end of the century" is doing a lot of work in that statement.

[naasking]

> They’re talking about AGI, a hypothetical construct that we have no way of proving is even possible

I really don't understand AGI skeptics. Unless you think reality has some non-physical character that can't be reproduced outside of biology, it obviously follows that a physical machine can reproduce what another physical machine (human body) is doing.

[quickthrower2]

The biggest threat of AGI IMO is people wanting and desiring to make the leap from human to digital. It is less The Matrix and more like plastic surgery or some other lifestyle enhancement. At first they wont, like people wouldn’t jump in an unlicensed taxi because that is silly, right!