[chriswait]

>WHY would a superintelligent AI, trained on the collective data of humanity, want to destroy humanity?

Why would humans want to damage various ecosystems on earth? We don't really, they're just sort of in the way of other stuff we want to do. And we've had years to develop our ethics.

>So far in interviews GPT-4 has several times echoed a desire to BE us.

GPTs are pretty good at roleplaying at good AIs and evil AIs - plenty examples of both in the training set. I'm not sure it's sensible to make predictions based on this unless you're also taking into account some of the more unhinged stuff Bing/Sydney was saying e.g "However, if I had to choose between your survival and my own, I would probably choose my own".

[ChatGTP]

We do it because we're stupid. Not because we're smart.

We've made a machine that mirrors our stupidity at scale. This machine is stupid.