[ben_w]

> show me an agent that persists productively in a goal without stopping. Does not exist.

The stories about agents bankrupting their owners by running too long passed you by?

> LLMs run on gradient descent.

They were *trained on*, they don't run on it.

Know what else is? DNA. A/B testing. Capitalism. Democracy.

> The agent is looking for the most efficient way to halt.

No. They are looking to produce an answer most likely to get a high score on a rating system which itself is another AI, created either manually or by yet another AI but in both cases to approximate what the creators think is "good", which may or may not be what anyone else thinks is "good", hence Grok calling itself Mecha Hitler because Musk is an edgelord.

> AGI paperclip maximizer woukd likely recognize the absurdity of its goal and shut itself down.

Do billionaires ever get satisfied with how much money they have?

[morpheos137]

persists productively. if agent bankruots its owner or.itself.then that is not productive (who funds paper clip maximizer?)

an answer that takes forever has no score and answer that is arrived at quickly and is good enough scores well. agents are indeed looking for the fastest route to superficially satisfy their constraints.

whether billionaores get stified is imaterial to the fsct they still are constraint bound.

[ben_w]

> persists productively. if agent bankruots its owner or.itself.then that is not productive (who funds paper clip maximizer?)

So, your standard for how risky it is, is simply how competent it is? (And if so, is the paperclip maximiser scenario really it being "successful"?)

That's fine right up until the thing passes an unknown threshold, one which will only be visible in the rear view mirror.

This is a problem in two directions:

1. We have a trend of the maximum complexity of a tasks they can handle growing faster than Moore's law did at its peak.

2. The threshold can be quite small, e.g. the covid-19 virus itself is not what anyone would call smart, ditto HIV, smallpox, and bubonic plague, but a genome is much the same learning system, and they still killed millions each.

> superficially satisfy their constraints.

The "superficially" part is one of the reasons these things can be dangerous. e.g. hopefully nobody at OpenAI actually wanted their wildly-sycophantic version, but yet they created it.

This is in fact the whole reason for the paperclip maximiser scenario: some idiot specifies the constraint "maximise paperclips" and it (in the hypothetical) superficially satisfies this without any consideration of why someone might ask for it.

[morpheos137]

persists productively. if agent bankruots its owner or.itself.then that is not productive (who funds paper clip maximizer?)

an answer that takes forever has no score and answer that is arrived at quickly and is good enough scores well. agents are indeed looking for the fastest route to superficially satisfy tjeir constraints.

whether billionaores get stified is imaterial to the fsct they still are constraint bound.