Not OP, but I do work in reducing harms of deployed AI systems at a big tech company
When most EA/Rationalist folks discuss AI Safety. They’re talking about AGI, a hypothetical construct that we have no way of proving is even possible
Now there’s nothing inherently wrong with funding research of that sort.
The problem comes is that many EA/Rationalist folks begin downplaying legitimate tangible risks on the horizon (I’ve had prominent EA folks tell me that Climate Change is not an existential risk) or that many of these AI Safety researchers actively ignore how to reduce harms of the AI we have deployed today.
This existential measuring contest is just so odd and unnecessary. I really don’t care if you think AI Safety is a legitimate problem. I personally don’t but it’s when they begin to downplay others actual catastrophes that many of us take issue.
The atom bomb was once a hypothetical construct. The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible. Proof is irrelevant outside of mathematics; the only relevant question is cost to expected benefit - or expected harm.
With dangerous research, when a conclusive demonstration of possibility is achieved, it is usually too late to do much in the way of prevention. That's why we don't expect the FDA to "prove" that harm from a new medication "is even possible" before regulating it; rather, we expect the pharmaceutical company to provide evidence for its expected harmlessness before we even make the attempt. Think of the FDA what you like, this is certainly a better standard than putting the burden of proof on harmful outcomes.
Part of the problem with this is that conclusive demonstration that AGI is possible is not even in the same ballpark as a conclusive demonstration that it would have consequences like the EA folks seem to believe.
From everything I've read about their views, they seem to believe that either the moment an AGI is created or very, very soon thereafter, it will achieve the Singularity and "ascend" to something near godhood from our perspective, and it will perceive humanity as a threat to it and immediately seek our destruction—which, because of its near-godhood, it will be able to achieve far faster than we have the ability to respond to. Thus, what we have to do is either prevent AGI from ever coming to pass (because as soon as it exists, it's effectively too late), or make absolutely certain that by the time it does, we have strong measures in place to either fight back against it, or get humanity the hell out of Dodge.
This is built on such a foundation of shaky assumptions that giving it credibility is ludicrous.
Plainly, that is not the case, as you are choosing to not spend your every waking moment to stop the antichrist AI coming into existence.
So either it's less obvious than you're claiming, or you are taking the threat of creating an evil AGI not seriously enough according to your own beliefs.
I don't spend every waking moment stopping lots of things that could kill us all. Nuclear war, climate change, gain-of-function research.
I just don't care that much about things that kill everyone. I don't think almost anyone does. In the Cold War, the vast majority of the population was not doing absolutely everything to achieve bilateral disarmament. This would lead you to believe that they didn't really expect nuclear war. But maybe they were just apathetic and helpless.
The atom bomb was once a hypothetical construct. The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible.
This isn't an accurate representation, really. The letter was written exactly because it was clear that a fission bomb is very likely within the short-term reach of current technological and industrial capability. This is very much not the case with AGI.
The disagreement is about the difficulty of "proving". I could say that unfriendly AI is clearly possible because evil humans exist. You could say that unfriendly AI is not proven possible until one has actually been built. But if you applied that standard to the letter, it would likewise fail it. Now, I agree that the case for AI is weaker, maybe much weaker, than the case for nuclear fission when the letter was written - there is, for instance, no clear research agenda that is held by preeminent researchers in the field to lead to it - though there are research groups, such as Deepmind, that have AGI as their explicit goal - but that also has nothing to do with being "proven" possible.
My argument is that for contested and dangerous outcomes, proof of possibility is an unreasonable standard, in good part because it massively underspecifies the goalposts.
Is there an accepted unambiguous definition of "existential" here?
Is it:
1. All life on the planet dies
2. All advanced life on the planet dies?
3. All humans die?
4. Advanced civilization is destroyed irretrievably?
5. Advanced civilization is destroyed for a long period of time?
etc
And I'm not sure the group "people who know anything about climate change" are neccesarily better equipped to answer this question - it's a complex systems question.
I'm not actually sure who is equipped to answer it accurately.
I'm not sure if this would actually count as "existential", but I think that it's well worth being very, very concerned about a consequence on the order of
"Millions to billions of people die; large percentages of the rest are displaced and/or have their lives made significantly shorter, harsher, and less certain as war, famine, disease, and natural disasters become vastly more common around the globe."
And there's really very little question that that's where climate change is leading if we don't get it under control soon.
> The problem comes is that many EA/Rationalist folks begin downplaying legitimate tangible risks on the horizon (I’ve had prominent EA folks tell me that Climate Change is not an existential risk) or that many of these AI Safety researchers actively ignore how to reduce harms of the AI we have deployed today.
I'm sure some people do this, and it's bad.
But to be clear, when you say some researchers are actively ignoring how to reduce harms of AI today... I'm not sure what you mean by "actively" ignore, but don't most researchers just by default ignore this, except for the few who are active in this field?
In other words, I think it's totally legit to focus on long-term AGI risk, and totally ignore current short-term AI risk, as a personal career-choice, just like it's ok to not even work on AI at all. Unless someone is actively trying to cause less resources to go into short-term AI risk, what's the problem?
> They’re talking about AGI, a hypothetical construct that we have no way of proving is even possible
I mean, we exist as (one would hope) intelligent beings. Why wouldn't AGI be possible? Your frame makes it seem like the default is that it isn't possible, which is wrong IMO.
They are right. Climate change is absolutely not an existential risk.
“In a high-emissions scenario where little is done to curb planet-heating gases, global mortality rates will be raised by 73 deaths per 100,000 people by the end of the century”
> Climate change is absolutely not an existential risk.
That's an assessment based on what we know, but there's a lot we don't. For instance, it's possible that climate change could lead to a large scale collapse of food chains that could pose an existential risk to humanity.
It's also plausible that climate change will lead to resource shortages that result in wars between nuclear powers, and the nuclear calamity that follows kills us all.
> They’re talking about AGI, a hypothetical construct that we have no way of proving is even possible
I really don't understand AGI skeptics. Unless you think reality has some non-physical character that can't be reproduced outside of biology, it obviously follows that a physical machine can reproduce what another physical machine (human body) is doing.
The biggest threat of AGI IMO is people wanting and desiring to make the leap from human to digital. It is less The Matrix and more like plastic surgery or some other lifestyle enhancement. At first they wont, like people wouldn’t jump in an unlicensed taxi because that is silly, right!
At this point I think that giving $5 to a homeless person in SF (even if they go straight to doing another shot) might be more productive than this self-aggrandizing vapid BS about some theoretical risk that might come in the future
Right now what's threatening the most people is war, famine and diseases. Climate change is a tangible future threat but I'd bet a couple $ that the JSO organizers are full EA proponents and are flying private jets to convince people to throw soup into works of art
> Like increased rate of hurricanes due to climate change?
This is a bad analogy. The mechanisms and risks of climate change are reasonably well understood as are the solutions available to us now and even some possible future solutions.
When it comes to AGI, what's the risk? Something bad might happen? The mechanisms of AGI arising is also unknown since it has never happened before and it's not even known if it's possible. Finally, what's the solution? Prevent AGI research? Make sure every AGI project has a killswitch? How do you prevent something when you don't know if it is possible nor how it will arise?
Compare this to the concrete, measurable things we can do to slow down climate change and "AGI safety" seems like a make-work money pit.
I'm not the poster but there is no reason to believe that we possess the tools to contemplate AI safety worse it may be a fools errand to contemplate how to control something much smarter than you are.
Ergo at this point in time money spent to pay people to contemplate it is almost certainly entirely wasted.
Interestingly, the sun would kill us much sooner than that: in less than 1 billion years the sun's increased output will heat up the Earth's surface to the point of becoming unlivable.
Related to the topics of AI apocalypse and putting things off because they seem far away: I personally find it amusing to contemplate a strong AGI being developed in 2035 and wiping out humanity. Why? Because it would eliminate the Y2038 problem, retroactively vindicating all the engineers who only allocated 32 bits to timestamps. It would mean the justification of "oh, it's so far off, the world will have changed unimaginably by then", would for once turn out to be correct!
Current AI seems to be just a box of math churning out correlations that we can put to effective use but not understand wherein the ability to actually act is entirely reliant on us deliberately wiring the machine into the world in a way it can even have negative effects or any effect at all wherein said negative effects are entirely and trivially predictable and avoidable. This is true whether its a robot that accidentally breaks a kids finger or a system that bakes in the inherent racism of prior decisions into future sentencing or mortgage approvals. In most cases the logical thing is just don't use AI for that.
Its not at all clear that future actual AI will be GPT7 now with 10,000x as much processing power. In fact not much seems to be clear at all. It would seem that the insight to control future tools will come from the experience in building said tools with little chance that it will just accidentally suddenly become skynet. If we doom ourselves thus it will be a long laborious process with years of striving, setbacks, and thousands of people involved.
We should focus on the basic science and let it become clear what avenues exist over the probable decades it will take to even reach the vicinity of our goal. Money spent specifically on paying PhDs to imagine how to secure a technology we don't have and don't remotely understand is probably wasted.
When most EA/Rationalist folks discuss AI Safety. They’re talking about AGI, a hypothetical construct that we have no way of proving is even possible
Now there’s nothing inherently wrong with funding research of that sort.
The problem comes is that many EA/Rationalist folks begin downplaying legitimate tangible risks on the horizon (I’ve had prominent EA folks tell me that Climate Change is not an existential risk) or that many of these AI Safety researchers actively ignore how to reduce harms of the AI we have deployed today.
This existential measuring contest is just so odd and unnecessary. I really don’t care if you think AI Safety is a legitimate problem. I personally don’t but it’s when they begin to downplay others actual catastrophes that many of us take issue.