[root_axis]

> Both OpenAI and Anthropic were founded by people who sincerely believed in the risk of out-of-control superintelligence.

They don't. LLMs can never become out-of-control superintelligence and everyone working on LLMs knows this (with a few eccentric exceptions).

[ben_w]

> LLMs can never become out-of-control superintelligence

The "never superintelligence" part I'll buy, though only in the sense of sample efficiency and generalisation ("quality superintelligence"), as they clearly have a superhuman breadth of skills, and run at superhuman speed.

"Never" out-of-control is obviously falsified by the already existing headlines about times they've gone out of control… in part, in some cases, because of their superhuman speed.

[root_axis]

They've never gone out of control. All those headlines are cases where humans deliberately relinquished control.

[ben_w]

A distinction without a difference.

If you're asleep at the wheel, you're legally responsible for the car crash, but the car itself was the thing which by crashing caused injuries.

If you deliberately relinquished control of your computer to OpenClaw, you're (I hope) legally responsible for whatever it does, but that doesn't stop it bankrupting you or deleting all your emails or whatever it was you connected it to.

DNA printers exist and are a thing. In 2010 we could all tell ourselves that no sane person would ever let some future AGI "out of the box" and onto the internet. Today, with the benefit of hindsight, do you seriously expect nobody to connect an LLM to a DNA printer, despite this being a terrible idea, given all the other things they've connected LLMs to despite it being a terrible idea?

[root_axis]

> A distinction without a difference

That's absurd. The distinction is at the heart of the entire discussion.

It's fine if you want to discuss the disruptive effects of LLMs in the hands of the masses, but that's not what anyone means when they say "out-of-control" in the context of the ASI meme

[ben_w]

> It's fine if you want to discuss the disruptive effects of LLMs in the hands of the masses, but that's not what anyone means when they say "out-of-control" in the context of the ASI meme

On the contrary.

The e.g. paperclip maximiser isn't "an AI decides to make paperclips", it is "some idiot tells an AI to make paperclips and leaves it unsupervised".

Even when people were priding themselves on plain just not believing Yudkowsky's claims that him role-playing as an AI could talk people into letting him out of the box*, the entire point was "let's get an AI to do work so we don't have to".

The entire point of AI has always been to automate stuff, to let ourselves not have to think about the stuff it does. Same as industrial machinery, and it took us long enough to sort out workplace health and safety and emergency off-switches for those. Or even more basic things like not having kids dart in and out of the unstoppable moving steam looms while they were in motion.

We are really, really slow at safety for this kind of thing.

* https://www.yudkowsky.net/singularity/aibox

[ux266478]

Which is going out of control. Something not under control is out of control. If I jump out of a moving car, I deliberately relinquished control of it. It's still out of control. What a silly semantic game.

[hollerith]

"LLMs can never become out-of-control superintelligence" might be relevant if there weren't many hundreds of researchers working (at OpenAI, Anthropic and elsewhere) on AI designs not based on the transformer (LLM) architecture.

[root_axis]

People are working on lots of things all the time, so far, nothing has approached the efficacy of the transformer architecture.

LLMs didn't emerge by chance, they are the culmination of decades of research intersecting with brute force engineering rigor in a perfect storm of innovation. You're not just going to stumble into an alternative approach by dumping loads of cash into research.

[hollerith]

Most of those researchers would disagree with your statement that the next major breakthrough is probably decades in the future.

Many of the insights accumulated in the decades up to now will probably prove useful for creating non-LLM AI, and the researchers can use LLM AI to speed up their research into non-LLM AI.