[janalsncm]

It’s not that AGI wouldn't be a problem. It’s that it sucks all of the oxygen out of the room when instead people should be talking about the problems that automated systems are causing right now. Is it not an existential threat that large swaths of the population will essentially become redundant in the next decades?

Deindustrialization took 50 years in the US. Imagine if it took 5 or 10, how much worse it would be. Or imagine hundreds of millions of people in China being cut off at the knees because their jobs were automated or reliant on a person who is no longer necessary.

Is it not an existential threat to democracy when trillion dollar multinational companies suck up all of the productivity gains and begin to manipulate the very systems which are supposed to regulate them? Strictly speaking, corporations are collective superintelligences as defined by Nick Bostrom.

[zygi]

> Is it not an existential threat that large swaths of the population will essentially become redundant in the next decades?

This is precisely the crux. In the scenario where there's no mass-paperclipping etc, AI is not an existential threat. It's a threat to the current regime, and I probably agree with you that the regimes it enables will be worse for people with our current values, but it's not an existential threat to humanity.

The people who worry about AI existential threats primarily worry about one thing: can humanity as a species survive? There are likely scenarios where it won't. Those scenarios should be prevented.

Another frame to think about this is: a libertarian probably wouldn't be concerned about the threats described in your comment. But both a libertarian and a socialist will be worried about paperclipping.

[dTal]

This has a Pascal's wager sort of feel. The possible negative consequences of a rampant AI are infinite, so we all ought to take it Very Seriously. The trouble is that there are lots of other scenarios with unbounded negative value - for instance, tight centralized control of technology supports a stable dystopia that keeps nearly all humans in a perpetual state of suffering. It's not at all clear, when you add it all up, that "prevent 'misaligned' AIs from being deployed at all costs" is the correct strategy to minimize risk.

[circuit10]

The difference to Pascal’s wager is that with that the probability is vanishingly small, but given current progress the probability of AI causing an existential threat is actually quite high. You can’t just call anything that has very bad negative consequences Pascal’s wager

Video about this: “Is AI Safety a Pascal’s Mugging?” https://youtu.be/JRuNA2eK7w0

[circuit10]

I sort of mixed up “wager” and “mugging” here but you get the point

[dTal]

Not at all, "Pascal's mugging" was exactly the term I would have used, had I remembered it :) That video is a very good reply. One response that comes to mind is that there is an implicit subtext behind any discussion of AI safety that we must do something about it, and soon. A good "anti-god" for his payoff matrix would be "the act of calling for AI safety will inadvertently be used as political capital by those wishing to be the gatekeepers of powerful technology" - the chances of that are quite high, and the negative consequences quite bad, if not necessarily apocalyptic.

[mario-main]

What is paperclipping? I looked it up and the results were all about a dating trend.

[cratermoon]

"The paperclip maximizer is a thought experiment described by Swedish philosopher Nick Bostrom in 2003. It illustrates the existential risk that an artificial general intelligence may pose to human beings were it to be successfully designed to pursue even seemingly harmless goals and the necessity of incorporating machine ethics into artificial intelligence design"

https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...

[sverona]

It's a reference to the paperclip maximizer.

[more_corn]

This is a false choice fallacy.

We’re perfectly capable of simultaneously acknowledging two risks. Imagine a toxic substance that is both an inhalation hazard and a burn hazard. You’d never caution people to stop talking about the inhalation hazard “because it sucks all the oxygen out of the room” and masks the burn hazard. You address both hazards at the same time.

There are multiple risks inherent in AI research. It’s ok to acknowledge them all. Some people assert that the existential risk is long term and unlikely so we should focus on the immediate risk. That is a mistake because the existential risk is not long term and not unlikely.