Read up about the "alignment problem". It is one that ai researchers cannot Crack and its literally what you wrote. The idea that we could never in any way define goals align with us.
We have been living with this problem of poorly defined goals for ages. Computers do what they are told to, not what we want them to. I built a career on translating what people want computers to do into what computers need to be told.
I suspect we'll hear of Dr. Susan Calvin more and more in the future.
I already know about the efforts about AI alignment - this is the reason I'm afraid about natural stupidity. I really do not want to see the alignment issue solved in my lifetime.
I suspect we'll hear of Dr. Susan Calvin more and more in the future.