[peterwwillis]

Have any of these tech "visionaries" (aka millionaires that we pretend have more insight than non-millionaires) considered that at the same time that we've increased our wealth we've also done more damage to our environment? If the cost of increased wealth is decreased habitat, will peak wealth result in the death of nature?

Aside: people are still comparing AI to a machine that feels, and this is so stupid it boggles the mind. The machine that creates paper clips will not kill off anyone who tries to turn it off because it does not have self-preservation, which is a system dependent on fear of death. Machines do not fear, and even if they did they would not fear death, because they have solid state. Bio systems are walking RAM disks.

Algorithms are basically math problems. The financial system and government do not work based on math problems, they work based on emotional instability. Seriously. Both these systems are driven almost entirely by the feelings of humans. They aren't algorithms, they are shitty biological systems that don't make mathematical sense at all.

[victorNicollet]

If a good algorithm optimizes for "quantity of paperclips produced", then it would recognize that "prevent humans from turning me off right now" is an optimal strategy. No fear of death involved, only pure rational optimization.

[peterwwillis]

The thought experiment is not realistic, because no intelligent system would think it could produce paperclips infinitely, and it would already have an "off feature" which was designed for a purpose (maintenance, update, replacement), and was intended to be used. An intelligent system would not reject proper use.

We already replace humans at their jobs all the time and they only very rarely kill their masters to prevent it.

[ric2b]

> because no intelligent system would think it could produce paperclips infinitely

This is irrelevant, being turned off while it's still possible to produce more paperclips would not be the optimal strategy for the machine trying to maximize paperclip production.

> and it would already have an "off feature" which was designed for a purpose

Look up what the control problem is. We don't know how to design such a feature for a General AI that also lets it do it's job effectively. It's not as simple as it looks. If you're not talking about a General AI than sure, it's easy, but non-General AI's are not very scary.

> We already replace humans at their jobs all the time and they only very rarely kill their masters to prevent it.

But humans don't have "maximize my production for this company" as their main life objective. Things like food, not going to jail, not dying, etc usually come first.

[peterwwillis]

kosta_self: I think you're hellbanned.

Good point, but that would be a bug in the AI. An intelligent system would see there would be no point in killing humans, so killing the humans would defeat the purpose of making the paperclips, and the humans support the machine, thus killing them would be counter-productive to its purpose.

It's also unrealistic that algorithms would try to exceed the limitations of their system. Imagine if a natural predator tried to "maximize production" of killing its prey: it would run out of prey, starve, and die. An AI would understand that trying to "maximize production" to the detriment of everything else would be counter-productive and create a resource contention war. Humans are the only creature I know of that exhausts natural resources to its own detriment - we are smart enough to exploit our resources, but too stupid to know when to stop. The natural system's response to this behavior seems to be to get us to kill ourselves. Maybe the AI are part of this process?

[kosta_self]

> The machine that creates paper clips will not kill off anyone who tries to turn it off because it does not have self-preservation

The idea here is that the algorithm tries to maximize its paperclip production and turning the machine off will definitely have a negative impact on its paperclip production. So the machine will take the most efficient measure to stop that negative impact.

[jam4peaches]

with autonomous drones and AI warfare, say goodbye to the birdies. chomp