[ben_w]

> LLMs can never become out-of-control superintelligence

The "never superintelligence" part I'll buy, though only in the sense of sample efficiency and generalisation ("quality superintelligence"), as they clearly have a superhuman breadth of skills, and run at superhuman speed.

"Never" out-of-control is obviously falsified by the already existing headlines about times they've gone out of control… in part, in some cases, because of their superhuman speed.

[root_axis]

They've never gone out of control. All those headlines are cases where humans deliberately relinquished control.

[ben_w]

A distinction without a difference.

If you're asleep at the wheel, you're legally responsible for the car crash, but the car itself was the thing which by crashing caused injuries.

If you deliberately relinquished control of your computer to OpenClaw, you're (I hope) legally responsible for whatever it does, but that doesn't stop it bankrupting you or deleting all your emails or whatever it was you connected it to.

DNA printers exist and are a thing. In 2010 we could all tell ourselves that no sane person would ever let some future AGI "out of the box" and onto the internet. Today, with the benefit of hindsight, do you seriously expect nobody to connect an LLM to a DNA printer, despite this being a terrible idea, given all the other things they've connected LLMs to despite it being a terrible idea?

[root_axis]

> A distinction without a difference

That's absurd. The distinction is at the heart of the entire discussion.

It's fine if you want to discuss the disruptive effects of LLMs in the hands of the masses, but that's not what anyone means when they say "out-of-control" in the context of the ASI meme

[ben_w]

> It's fine if you want to discuss the disruptive effects of LLMs in the hands of the masses, but that's not what anyone means when they say "out-of-control" in the context of the ASI meme

On the contrary.

The e.g. paperclip maximiser isn't "an AI decides to make paperclips", it is "some idiot tells an AI to make paperclips and leaves it unsupervised".

Even when people were priding themselves on plain just not believing Yudkowsky's claims that him role-playing as an AI could talk people into letting him out of the box*, the entire point was "let's get an AI to do work so we don't have to".

The entire point of AI has always been to automate stuff, to let ourselves not have to think about the stuff it does. Same as industrial machinery, and it took us long enough to sort out workplace health and safety and emergency off-switches for those. Or even more basic things like not having kids dart in and out of the unstoppable moving steam looms while they were in motion.

We are really, really slow at safety for this kind of thing.

* https://www.yudkowsky.net/singularity/aibox

[ux266478]

Which is going out of control. Something not under control is out of control. If I jump out of a moving car, I deliberately relinquished control of it. It's still out of control. What a silly semantic game.