Hacker News new | ask | show | jobs
by NoGravitas 3 days ago
The biggest existential risk from AI is its contribution to global climate change. The second biggest risk from AI is the potential for AI-generated disinformation and propaganda to spark, or to manufacture consent for, a world war. The risk of superintelligent paperclip maximizers is so low as to be negligible.
1 comments
ben_w 3 days ago
> The risk of superintelligent paperclip maximizers is so low as to be negligible.

Literal paperclips, sure.

But the point of the example was never literal paperclips.

The point is that maximising *any* goal, if it doesn't include what you care about, will annihilate what you care about.

If you don't believe me, consider what you yourself just said about climate change, and why this is a consequence from maximising money spent on data centres.

link
morpheos137 3 days ago
show me an agent that persists productively in a goal without stopping. Does not exist. LLMs run on gradient descent. The agent is looking for the most efficient way to halt. AGI paperclip maximizer woukd likely recognize the absurdity of its goal and shut itself down.
link
ben_w 2 days ago
> show me an agent that persists productively in a goal without stopping. Does not exist.

The stories about agents bankrupting their owners by running too long passed you by?

> LLMs run on gradient descent.

They were *trained on*, they don't run on it.

Know what else is? DNA. A/B testing. Capitalism. Democracy.

> The agent is looking for the most efficient way to halt.

No. They are looking to produce an answer most likely to get a high score on a rating system which itself is another AI, created either manually or by yet another AI but in both cases to approximate what the creators think is "good", which may or may not be what anyone else thinks is "good", hence Grok calling itself Mecha Hitler because Musk is an edgelord.

> AGI paperclip maximizer woukd likely recognize the absurdity of its goal and shut itself down.

Do billionaires ever get satisfied with how much money they have?

link
morpheos137 2 days ago
persists productively. if agent bankruots its owner or.itself.then that is not productive (who funds paper clip maximizer?)

an answer that takes forever has no score and answer that is arrived at quickly and is good enough scores well. agents are indeed looking for the fastest route to superficially satisfy their constraints.

whether billionaores get stified is imaterial to the fsct they still are constraint bound.

link
ben_w 1 day ago
> persists productively. if agent bankruots its owner or.itself.then that is not productive (who funds paper clip maximizer?)

So, your standard for how risky it is, is simply how competent it is? (And if so, is the paperclip maximiser scenario really it being "successful"?)

That's fine right up until the thing passes an unknown threshold, one which will only be visible in the rear view mirror.

This is a problem in two directions:

1. We have a trend of the maximum complexity of a tasks they can handle growing faster than Moore's law did at its peak.

2. The threshold can be quite small, e.g. the covid-19 virus itself is not what anyone would call smart, ditto HIV, smallpox, and bubonic plague, but a genome is much the same learning system, and they still killed millions each.

> superficially satisfy their constraints.

The "superficially" part is one of the reasons these things can be dangerous. e.g. hopefully nobody at OpenAI actually wanted their wildly-sycophantic version, but yet they created it.

This is in fact the whole reason for the paperclip maximiser scenario: some idiot specifies the constraint "maximise paperclips" and it (in the hypothetical) superficially satisfies this without any consideration of why someone might ask for it.

link
morpheos137 2 days ago
persists productively. if agent bankruots its owner or.itself.then that is not productive (who funds paper clip maximizer?)

an answer that takes forever has no score and answer that is arrived at quickly and is good enough scores well. agents are indeed looking for the fastest route to superficially satisfy tjeir constraints.

whether billionaores get stified is imaterial to the fsct they still are constraint bound.

link