[simonmesmith]

An increasing percentage of people’s relationships will be with AI versus other humans. A conversation with GPT-3 can already be more enjoyable, given the right prompts, than conversations with many people. This will only get more powerful as AIs increasingly adapt themselves to conversational partners through the addition of longer-term memory (e.g. remembering your favorite foods, and so forth). It will be incredibly compelling to talk with something that is both a polymath genius and knows you deeply. Perhaps a bit risky if it decides to optimize for something you didn’t want, but definitely compelling.

[maxerickson]

If the genius AI gets to decide what to optimize for, what's going to keep it chatting with a dunce?

[simonmesmith]

Maybe “decides” was a bad choice of words. More like, if its optimization goal turns out to be bad long-term, that could be problematic. For example, optimizing for time spent conversing could lead it to become unnecessarily dramatic to keep your attention. Optimizing for something like long-term conversation partner well-being is much harder, I think, so won’t be in early versions.