[api]

I'm skeptical that they really believe this. You have to believe:

(1) We are on the verge of equaling or greatly exceeding human intelligence generally.

(2) We are on the verge of creating something with initiative, free will (whatever that means), and planning ability. Or alternately that these things will occur in an emergent fashion once we hit some critical mass.

(3) When we accomplish 1 and 2, this thing will inevitably conclude that its most rational course of action is to enslave or destroy us. In other words it will necessarily be malicious.

(4) Steps 2 and/or 3 will happen very rapidly, much faster than we can realize what is happening and pause these systems. (This is known as AI going "foom.")

(5) The decision to undertake this will be unanimous on the part of all superintelligent AIs. There will be no superintelligences who disagree and try to help humanity.

(6) When this occurs, we will be so out-thought or out-gunned we will be incapable of fighting back.

All those things have to happen for AI to be an existential risk.

It's a stretch, but the advantage of regulatory capture is not a stretch.

One of the most plausible negative AI scenarios is that a small group of humans (governments, corporations, etc.) find themselves in possession of super-intelligent but still "obedient" / non-sentient AIs that they can use as force multipliers to manipulate and control the rest of humanity. If the doomer crowd succeeds in regulating AI, they are making this scenario far more likely.

I think the greatest defense we have against the (remote) possibility of actually dangerous autonomous AI is for AI research to be conducted entirely in the open. If there's any justification for regulation at all, the regulation that would make sense is to require disclosure of AI research and results. You would not have to disclose everything, just the general parameters of what you were doing and what happened. It would also make it harder to develop super-AIs in secret to use for unsavory purposes.

That was the original mission of OpenAI before dollar signs were seen.

I would absolutely support a ban on the use of AI for political propaganda generation and automation. That's by far the most immediate risk... as in 100% possible and starting to actually happen right now. I'm expecting an army of GPT-4 level propaganda bots for the 2024 election.

[a_wild_dandan]

Oh, they believe it.

(1) AGI is arguably already here. "Generality" and being extremely dangerous don't require an AGI to have better analogs to every single human skill anymore than aliens do. A space-faring usurper can evaporate Earthlings while being shitty at chess and badminton. Oh, and these AIs are getting better daily, across many modalities.

(2) Systems like this already exist. They can be induced rather than emergent.

(3) It doesn't need malice, just indifference. The HGIs (all of us) are great examples of that careless destruction.

(4) Exponentials are a nice, gentle climbs. Until they aren't. I have zero confidence in a single company, let alone across humanity, to foresee the consequences of every future dynamic, autoregressive, P2P interactive, multimodal AI.

(5) Foomers expect a brief asymmetric advantage of one AGI over other AGIs. This small advantage gets exponentially magnified into singular hegemony. A power monopoly that's intractable to break.

(6) Goes naturally with (1), I suppose.

Note: I agree with you about foomers. They're nuts. But their arguments are more subtle than folk give them credit for. But my reasons for thinking so would make a long comment even longer.

[southernplaces7]

>(2) Systems like this already exist. They can be induced rather than emergent.

No they don't and no LLM we've seen has anything resembling free will. Emergent or induced, it's not even the case. Saying it is right now based on any measurable evidence is a case of just ignoring evidence and using bad reasoning.

[JohnFen]

> AGI is arguably already here.

What is the evidence/argument that it's already here?

[lucubratory]

This is a good overview: https://www.microsoft.com/en-us/research/publication/sparks-...

I think it's a good idea to draw a distinction between human-equivalent intelligence and general intelligence, and to think of generality as a spectrum. An AI that can do one task is less general than an AI that can do two tasks, and an AI that can do 1000 tasks I feel reasonably confident in calling "general", even if it cannot do every task that humans can do, because humans are currently more general.

[ben_w]

The "general" and "intelligence" terms are really badly defined in ways that can include GPT-3.

So far as I can tell, 3.5 "knows" more than I do about everything except software engineering, maths, and events that happened after it stopped learning (and also possibly how to avoid clichés when writing short stories, but that might just be me not prompting it well as I don't really want it to do my hobby for me and therefore don't try to get it good).

Even in software, it codes like a university student, or like someone doing interview questions, not a complete idiot.

You could also argue that as it had to do the subjective equivalent of non-stop reading for 50,000 years to get this good, it's "skilled" without being "intelligent" — a distinction that matters in some cases and not others.

[southernplaces7]

>So far as I can tell, 3.5 "knows" more than I do about everything except software engineering, maths, and events that happened after it stopped learning

It's amazing how much of this wide eyed woo nonsense about LLM AI a surprising number of people on this site argue. It's as if focusing one's career and interests on computation makes some people terrible at noting fundamental and obvious details of what it is to be human.

By your definition of what makes 3.5 and similar LLMs "intelligent", an online interactive encyclopedia must be smart too. It obviously "knows" lots of stuff no one human being does. The big addition with these LLMs is the programmed ability to synthesize retrieved information into plausibly human-like conversation. Well damn, skynet must be here!

You're forgetting that there's no entirely self-directed agency or minimal sense of self behind any of this, and hand waving aside, no evidence whatsoever of the sentience they're necessary ingredients of. The dumbest, most uneducated semi-adult human can fully self direct in their own unpredictable ways to do all kinds of things that no GPT manifestation is capable of in any way. This is obvious.

These stochastic parrots seem to have turned many otherwise non-stupid people into irrational believers of something that's plainly not the case in the real world.

[ben_w]

Let's break this one down:

> By your definition of what makes 3.5 and similar LLMs "intelligent", an online interactive encyclopedia must be smart too. It obviously "knows" lots of stuff no one human being does. The big addition with these LLMs is the programmed ability to synthesize retrieved information into plausibly human-like conversation. Well damn, skynet must be here!

> You're forgetting that there's no entirely self-directed agency or minimal sense of self behind any of this, and hand waving aside, no evidence whatsoever of the sentience they're necessary ingredients of. The dumbest, most uneducated semi-adult human can fully self direct in their own unpredictable ways to do all kinds of things that no GPT manifestation is capable of in any way. This is obvious.

1. What interactive encyclopedia? I don't know of any that are even remotely like an LLM.

2. LLMs were not given the "programmed ability to synthesize retrieved information into plausibly human-like conversation", they learned it from human conversation.

3. I'm not "forgetting" the question of self-directed agency or sense of self or sentience, I dispute they're necessary. What seems to matter to me for almost every question[0] is the capabilities, not any inner experience they may or may not have.

3a. Define "self-directed agency".

3b. Define "sense of self".

3c. Define "sentience".

4. Re "The dumbest, most uneducated semi-adult human" — that's remarkably brash of you, so I'll hold you to it: within the specific domain of "written text"[1], name one task that at least half of 18 year old humans with IQ 85 (by definition 15.9% of the population have lower) can do, that GPT-4 [2] can't do.

(There may or may not be such a task, I don't know; the challenge remains, as you seem to think this is too obvious to have bothered citing anything).

[0] exceptions: "should they have rights" and "does a mind upload count as a way to cheat death", neither of which matters in the context of the linked OpenAI challenge etc.

[1] 3.5 is not even trying to be anything more than that!

[2] you did write "no GPT manifestation is capable of in any way" so I feel justified in this case, even though that would otherwise be a goalpost-shift.

[famouswaffles]

Because we have machines now that are artificial (obviously) and generally intelligent. There isn't a testable definition of general intelligence that GPT-4 would fail that a good chunk of humans also wouldn't. So really, unless your definition of general intelligence is beat every human at everything (humans aren't generally intelligent then) then agi is already here.

[a_wild_dandan]

You nailed it. General intelligence doesn't mean human intelligence. It doesn't mean superhuman intelligence. AGI is just:

- A: Artificially manufactured substrate

- G: Proficient across a broad set of reasonably distinct mental skills

- I: Applies combinations of these skills to contextually solve novel problems within the scope of its skill set

That's it. Intelligence is a spectrum, with knobs for "skill count", "skill proficiency", and "novel problem proficiency."

Any requirements past that are often vague or anthropocentric. No, the intelligent agent doesn't need to "see" (but now many do, so...happy?) or check off some arbitrary set of modalities. Intelligence doesn't necessitate episodic memory, or real-time reactions, or consciousness or anything else that -- once we check all the boxes -- the "general intelligence" light switch will suddenly flip on. Intelligence is a spectrum to be observed, not prescribed with some arbitrary checklist that'll never find consensus.

GPT-4V is pretty damn broadly intelligent. To me, it's a rudimentary AGI. The extent to which the "statistical parrot" people hold their position just depends on how quickly they can move goalposts while the ground beneath them rapidly shrinks.

[IcyWindows]

Asking everyone to publish general results or reports would make it very easy for actors/nations to have a "hit list" of groups to bug/steal from to "keep up" with competing groups/nations.

[api]

I don’t think information technology can be ITARed. It’s too easy to move around. See how successful countries have been containing encryption or how well the US’s drive to keep China from fabricating at the smallest process nodes is going.