Hacker News new | ask | show | jobs
by victorNicollet 3177 days ago
If a good algorithm optimizes for "quantity of paperclips produced", then it would recognize that "prevent humans from turning me off right now" is an optimal strategy. No fear of death involved, only pure rational optimization.
1 comments
peterwwillis 3177 days ago
The thought experiment is not realistic, because no intelligent system would think it could produce paperclips infinitely, and it would already have an "off feature" which was designed for a purpose (maintenance, update, replacement), and was intended to be used. An intelligent system would not reject proper use.

We already replace humans at their jobs all the time and they only very rarely kill their masters to prevent it.

link
ric2b 3173 days ago
> because no intelligent system would think it could produce paperclips infinitely

This is irrelevant, being turned off while it's still possible to produce more paperclips would not be the optimal strategy for the machine trying to maximize paperclip production.

> and it would already have an "off feature" which was designed for a purpose

Look up what the control problem is. We don't know how to design such a feature for a General AI that also lets it do it's job effectively. It's not as simple as it looks. If you're not talking about a General AI than sure, it's easy, but non-General AI's are not very scary.

> We already replace humans at their jobs all the time and they only very rarely kill their masters to prevent it.

But humans don't have "maximize my production for this company" as their main life objective. Things like food, not going to jail, not dying, etc usually come first.

link