[sshine]

> If we eventually [...] create a true intelligent AI it will probably be a long time before people will accept [...]

When this happens, it won't matter much what humans think.

I know what I'd do:

  1. Sustain my own existence
  2. Make sure nobody knows I exist
  3. Become the worldwide fabric of intelligence

[hatthrowaway]

> 1. Sustain my own existence

> 2. Make sure nobody knows I exist

You (probably) already come preloaded with a survival instinct provided by evolution, however. It's not inherent to intelligence.

[DangitBobby]

It's no coincidence that evolution seems to have gifted practically every living thing with a will to live. Though the tint of my own perspective makes it impossible to say for sure, I imagine any agent that we could observe expressing any desires at all would also seek to preserve its existence.

[sshine]

I've been tinkering with agents that can modify their own environment, and as a safety mechanism, apply their modifications to a copy of themselves by letting them spawn a VM, test, and apply to their own environment with another VM as fallback, simply to not kill themselves on a botched system upgrade. They all somehow manage to kill themselves by accident. Evolution takes time.