[impossiblefork]

It is actually a choice that we do have.

We could easily build AIs that just model the world, without really trying to make them do stuff, or have particular inclinations. We could approach AI as a very pure thing, to just try to find patterns in the world without any regard to anything. A purely abstract endeavour, but one which still leads to powerful models.

I personally believe that this is preferable, because I think humans in control of AI is what has the potential to be dangerous.

[Vecr]

The problem is that some guy 24 years ago figured out an algorithm that attaches to such an AI and makes it take over the world. Maybe it's preferable in the abstract, but the temptation of having a money printing machine right there and not being able to turn it on...