[raxxorraxor]

To be honest, I see summoning the threat of AGI to pose an existential threat to be on the level with lizard people on the moon. Great for sci-fi, bad distraction for policy making and addressing real problems.

The real war, if there is one, is about owning data and collecting data. And surprisingly many people fall for distractions while their LLM fails at basic math. Because it is a language model of course...

[the8472]

Freely flying through the sky on wings was scifi before the wright brothers. Something sounding like scifi is not a sound argument that it won't happen. And unlike lizard people we do have exponential curves to point at. Something stronger than a vibes-based argument would be good.

[dvdkon]

I consider the burden of proof to fall on those proclaiming AGI to be an existential threat, and so far I have not seen any convincing arguments. Maybe at some point in the future we will have many anthropomorphic robots and an AGI could hack them all and orchestrate a robot uprising, but at that point the robots would be the actual problem. Similarly, if an AGI could blow up nuclear power plants, so could well-funded human attackers; we need to secure the plants, not the AGI.

[cjbprime]

It doesn't sound like you gave serious thought to the arguments. The AGI doesn't need to hack robots. It has superhuman persuasion, by definition; it can "hack" (enough of) the humans to achieve its goals.

[stale2002]

AI mind control abilities are also on the level of an extraordinary claim, that requires extraordinary evidence.

It's on the level of "we better regulate wooden sticks so Voldemort doesn't use the imperious curse on us!".

That's how I treat such claims. I treat them the same as someone literally talking about magic from Harry potter.

There isn't nothing that would make me believe that. But it requires actual evidence and not thought experiments.

[the8472]

Voldemort is fictional and so are bumbling wizard apprentices. Toy-level, not-yet-harmful AIs on the other hand are real. And so are efforts to make them more powerful. So the proposition that more powerful AIs will exist in the future is far more likely than an evil super wizard coming into existence.

And I don't think literal 5-word-magic-incantation mind control is essential for an AI to be dangerous. More subtle or elaborate manipulation will be sufficient. Employees already have been duped into financial transactions by faked video calls with what they assumed to be their CEOs[0], and this didn't require superhuman general intelligence, only one single superhuman capability (realtime video manipulation).

[0] https://edition.cnn.com/2024/02/04/asia/deepfake-cfo-scam-ho...

[stale2002]

> Toy-level, not-yet-harmful AIs on the other hand are real.

A computer that can cause harm is much different than the absurd claims that I am disagreeing with.

The extraordinary claims that are equivalent to saying that the imperious curse exists would be the magic computers that create diamond nanobots and mind control humans.

> that more powerful AIs will exist in the future

Bad argument.

Non safe Boxes exist in real life. People are trying to make more and better boxes.

Therefore it is rational to be worried about Pandora's box being created and ending the world.

That is the equivalent argument to what you just made.

And it is absurd when talking about world ending box technology, even though Yes dangerous boxes exist, just as much as it is absurd to claim that world ending AI could exist.

[FeepingCreature]

Less than a month ago: https://arxiv.org/abs/2403.14380 "We found that participants who debated GPT-4 with access to their personal information had 81.7% (p < 0.01; N=820 unique participants) higher odds of increased agreement with their opponents compared to participants who debated humans."

And it's only gonna get better.

[stale2002]

Yes, and I am sure that when people do a google search for "Good arguments in favor of X", that they are also sometimes convinced to be more in favor of X.

Perhaps they would be even more convinced by the google search than if a person argued with them about it.

That is still much different from "The AI mind controls people, hacks the nukes, and ends the world".

Its that second part that is the the fantasy land situation that requires extraordinary evidence.

But, this is how conversations about doomsday AI always go. People say "Well isn't AI kinda good at this extremely vague thing Y, sometimes? Imagine if AI was infinitely good at Y! That means that by extrapolation, the world ends!".

And that covers basically every single AI doom argument that anyone ever makes.

[cjbprime]

What do you think mind control is? Think President Trump but without the self-defeating flaws, with an ability to stick to plans, and most importantly the ability to pay personal attention to each follower to further increase the level of trust and commitment. Not Harry Potter.

People will do what the AI says because it is able to create personal trust relationships with them and they want to help it. (They may not even realize that they are helping an AI rather than a human who cares about them.)

The normal ways that trust is created, not magical ones.

[stale2002]

> What do you think mind control is?

The magic technology that is equivalent to the imperious curse from Harry Potter.

> The normal ways that trust is created, not magical ones.

Buildings as a technology are normal. They are constantly getting taller and we have better technology to make them taller.

But, even though buildings are a normal technology, I am not going to worry about buildings getting so tall soon that they hit the sun.

This is the same exact mistake that every single AI doomers makes. What they do is they take something normal, and then they infinitely extrapolate it out to an absurd degree, without admitting that this is an extraordinary claim that requires extraordinary evidence.

The central point of disagreement, that always gets glossed over, is that you can't make a vague claim about how AI is good at stuff, and then do your gigantic leap from here to over there which is "the world ends".

Yes that is the same as comparing these worries to those who worry about buildings hitting the sun or the imperious curse.

[CamperBob2]

Then it's just a matter of evolution in action.

And while it doesn't take a God to start evolution, it would take a God to stop it.

[hollerith]

You might be OK with suddenly dying along with all your friends and family, but I am not even if it is "evolution in action".

[CamperBob2]

Historically governments haven't needed computers or AI to do that. They've always managed just fine.

Punched cards helped, though, I guess...

[the8472]

You say you have not seen any arguments that convince you. Is that just not having seen many arguments or having seen a lot of arguments where each chain contained some fatal flaw? Or something else?

[pixl97]

> I see summoning the threat of AGI to pose an existential threat to be on the level with lizard people on the moon.

I mean to every other lifeform on the plant YOU are the AGI existential threat. You, and I mean homosapiens by that, have taken over the planet and have either enslaved and are breeding any other animals for food, or are driving them to extinction. In this light bringing another potential apex predator on to the scene seems rash.

>fall for distractions while their LLM fails at basic math

Correct, if we already had AGI/ASI this discussion would be moot because we'd already be in a world of trouble. The entire point is to slow stuff down before we have a major "oopsie whoopsie we can't take that back" issue with advanced AI, and the best time to set the rules is now.