[zozbot234]

> We need an approach to make sure AI doesn't destroy the world and wipe humanity to extinction.

That's easy. Stop training your AIs on cheesy old sci-fi that talks about robot uprisings. In fact, maybe y'all should just stop talking about robot uprisings altogether. Putting a stochastic parrot in charge of an agentic function-calling REPL doesn't somehow make it super-dangerous, except to the extent that dumb mistakes might result in danger. And you can't prevent an AI from making dumb mistakes with burdensome regulation.

[scarmig]

Yes, we get that if you assume there is zero existential risk from AI, then there is zero existential risk from AI.

[NoGravitas]

The biggest existential risk from AI is its contribution to global climate change. The second biggest risk from AI is the potential for AI-generated disinformation and propaganda to spark, or to manufacture consent for, a world war. The risk of superintelligent paperclip maximizers is so low as to be negligible.

[ben_w]

> The risk of superintelligent paperclip maximizers is so low as to be negligible.

Literal paperclips, sure.

But the point of the example was never literal paperclips.

The point is that maximising *any* goal, if it doesn't include what you care about, will annihilate what you care about.

If you don't believe me, consider what you yourself just said about climate change, and why this is a consequence from maximising money spent on data centres.

[morpheos137]

show me an agent that persists productively in a goal without stopping. Does not exist. LLMs run on gradient descent. The agent is looking for the most efficient way to halt. AGI paperclip maximizer woukd likely recognize the absurdity of its goal and shut itself down.

[ben_w]

> except to the extent that dumb mistakes might result in danger

That "except" goes all the way up to starting WW3. Or a leak from a viral research lab, and by "leak" I mean "mail order" and by "research lab" I mean "the companies who already ship custom DNA and RNA retroviruses": https://duckduckgo.com/?q=companies+who+already+ship+custom+...

If you can prove that simply not training on horror stories would work, it would make a lot of people very happy.

Unfortunately, I don't think it does a single thing to solve, for example, Elon Musk just plain asking some future version of Grok to take over the world for him.

Nor would merely failing to include them in traing data stop certain entire fictional scenarios such as that Doctor Who episode where the android repair bots weren't told that the crew were off-limits as spare parts, or the other Doctor Who episode where the utilitarian robots started killing everyone who was upset because they calculated net positive utility from upset people ceasing to exist. Well, except for the bit where the Doctor saves the day, because they are not real.