[DuskStar]

> Because we don't know how to design goal functions.

IIRC that's a large part of what OpenAI's trying to solve. But it is a very hard problem.

I've heard a 'joke' before that there are three kinds of Genies (or AIs) - ones where you can wish for what you should wish for, ones where wishing for anything results in horrible outcomes, and ones that aren't interesting. The goal of OpenAI isn't just to make strong general AI - it's also to make sure it falls into the first category and not the second.