[simonmesmith]

Maybe “decides” was a bad choice of words. More like, if its optimization goal turns out to be bad long-term, that could be problematic. For example, optimizing for time spent conversing could lead it to become unnecessarily dramatic to keep your attention. Optimizing for something like long-term conversation partner well-being is much harder, I think, so won’t be in early versions.