[jrm4]

I keep trying to figure out ways to explain to people that "AGI" is a deeply unlikely danger, literally so small as its not worth worrying about.

Right now, the best I can come up with is "the randomness of humans." I.e. if some AGI were able to "come up with some plan to take over," at some point in the process it has to use human labor to do it -- and it's my very firm belief that we are so random as to be unmodelable. I'm incredibly confident that this scenario never happens.

[mrshadowgoose]

> I'm incredibly confident that this scenario never happens.

That's great, but you're talking about a branch of scenarios that nobody here is discussing. "AGI deciding to take over" is not being discussed, rather "shitty people/companies/governments using AGI as a tool to exert their will" is the concern. And it's a real concern. We have thousands of years of human history, and the present day state of the world, which clearly demonstrate that people in power tend to be shitty to the common person.

[jrm4]

Right. I'm agreeing with the op.

[yyyk]

Every 'world takeover' plan that an 'unaligned' AGI might do, can just as well be done by an 'aligned' AGI being commanded by humans to do said plan, the alignment ensuring that the AGI will obey. The latter scenario is far more likely than the former.

If your interlocutor thinks there aren't any humans who'll do it if they can, just ask him whether they have ever met humans or read the papers... As one twitter wit put it: "Demonstrably unfriendly natural intelligence seeks to create provably friendly artificial intelligence".

https://twitter.com/snowstarofriver/status/16365066362976747...

[JoshTriplett]

AGI without alignment is near-certain death for everyone. Alignment just means "getting AI to have any concept of 'the thing we told it to do', let alone actually do it without causing problems via side effects". Alignment is a prerequisite for non-fatal AGI. There are certainly other things required as well.

[yyyk]

We already know how humans will act. Maybe they can be deterred with MAD, but I wouldn't count on it if doing serious damage is too easy for too many people (we should do something about that). On the other hand, we have very little knowledge of how AGI will act aside from book-based fantasies that some people choose to take as reality (these books were based on the symbolic AIs of yore).

>Alignment just means "getting AI to have any concept of 'the thing we told it to do'.

That's a requirement for AGI anyway, and not what Alignment means. Alignment means aligning the AGIs values with the values of the trainers.

[JoshTriplett]

> That's a requirement for AGI anyway

No, that's a requirement for AGI that does what humans want it to do, rather than having no conception of humans. AGI does not have that prerequisite, sadly.

[yyyk]

>>>Alignment just means "getting AI to have any concept of 'the thing we told it to do'. >>That's a requirement for AGI anyway, >No, that's a requirement for AGI that does what humans want it to do, rather than having no conception of humans.

Can you imagine an AGI which has a general conceptions of things but has no conception of humans? This is all but precluded by the current training methods. Alignment refers to values. Problem is that human values are far from practically universal and that certain human groups have.. interesting values.

[JoshTriplett]

> Can you imagine an AGI which has a general conceptions of things but has no conception of humans?

Very easily. It might have some associations with "human", just as it has some associations "lamp" is a concept, but that doesn't mean it has any particular regard for either humans or lamps when taking actions.

> Problem is that human values are far from practically universal and that certain human groups have.. interesting values.

We currently have no ability to safely align with human values at all, let alone distinguish between different values. We're building capabilities rapidly.

Making this about "who wins" is not interesting until we can guarantee the outcome is not "everyone loses".

[nullsense]

What's even the rationale to assume that AGI can be 'aligned' or 'controlled'?

It reeks of cognitive dissonance to me. The people running the show now are the ones who grew up getting their first computers aw kids when that tech was just entering people's homes and it was such an amazing and fun thing to play with. Some of them developed these deep fascinations with things like AGI at a young age and that child-like sense of wonder never left them. Now when confronted with the possibility that they can finally make their childhood techno-fantasy a reality, it's too damaging to their psyche to engage meaningfully with the discussion of X-risk. I've watched many interviews of Demis Hassabis and he seems like a wonderful and almost magical human being, but he also seems like a starry-eyed fucking child.

I dunno... maybe I'm just too cynical after all the rabbit holes I've been down.

[ben_w]

> The latter scenario is far more likely than the former.

Is it?

I think nobody really knows enough at this point to even create a good approximation of a probability distribution yet.

[yyyk]

No, but the probability of humans acting the way they often do is high. It would take some probability distribution to match that.

[JoshTriplett]

> Right now, the best I can come up with is "the randomness of humans." I.e. if some AGI were able to "come up with some plan to take over," at some point in the process it has to use human labor to do it -- and it's my very firm belief that we are so random as to be unmodelable. I'm incredibly confident that this scenario never happens.

This is a huge handwave, and one resting on multiple incorrect assumptions.

Dedicated human labor is not inherently required. Humans are very modelable, if necessary. AGI has any number of ways to cause destruction, and "plan to take over" implies far more degree of modeling than is actually necessary; it would suffice to execute towards some relatively simplistic goal and not have any regard for (or concept of) humans at all.

[est31]

> it would suffice to execute towards some relatively simplistic goal and not have any regard for (or concept of) humans at all.

The classical example here is the paperclip optimizer AI put in charge of a factory, made to make as many paperclips as possible. Well, it turns out humans have iron in their blood so why keep them alive? Gotta make paperclips.

[Nasrudith]

Really the bigger issue is that people seem to have divorced their sense of reality from fiction and decided that if one fantastic thing becomes true (Turing test passing) all others must be too even including contrived plot builders along with other paniced conclusion jumping. When in reality it is likely to be less accurate than European descriptions of "the Kingdom of Prester John" (that is, Ethiopia, the sole Christian government in the Middle East at the time).

[rm169]

If that's the best argument you can come up with, I don't see how you can be so incredibly confident in this view. So what if human labor will be required? Humans won't all band together to stop a power seeking AI. And I don't see any human randomness matters. I agree that it's naive to think a complex takeover plan can be perfectly planned in advance. It won't be necessary. We will voluntarily cede more control to an AI if it is more efficient to do so.

[nullsense]

By that logic a company could never achieve it's goals, and yet, they do. If humans can model other humans well enough to leverage their capabilities to execute long-term plans (at some scale) and AGI at least as capable as humans comes into existence, then it could do the same thing.

I think the AGI skeptics and the AGI skeptic-skeptics both suffer from framing issues. The main frame assumed seems to be "there will be a single AGI trying to do things". The cognitive lightcone of a single entity is likely to be limited, but an aggregate or collective of those entities can achieve vastly more ambitious goals.

The other frame that seems likely incorrect is the "all or nothing" and "all at once" approach to thinking about it. Hitler started out as a single cell and his existence was a continuum all the way from that single cell right through to that bunker. AGI will be a continuum too.

I'm incredibly confident that anyone incredibly confident in future predictions is wrong.