[kkjjkgjjgg]

Would be a fun idea for a short story perhaps. An AI goes rogue trying to optimize its reward function, and humans lose hope to be able to stop it. In the last minute the AI figures out how to hack itself and enter the maximum reward, and mankind is saved another time.

[rescripting]

But what is the “maximum possible reward”? Does a limit exist? Or is it now consuming all possible resources to develop storage and compute resources to grow that limit…

[janto]

I imagine a paperclip factory with trucks driving in loops in front of a scanner that is over counting them as they drive past.

[ganzuul]

Deleting the reward function ends the game.

[kkjjkgjjgg]

It could also change the way its reward function is being computed.