[ctoth]

The concept you are describing is called Coherent Extrapolated Volition.

> In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.

[evanevan]

Thanks - yeah I think CEV hits the mark. Would be awesome if OpenAI (and others) committed to CEV then (with I'd add the qualification, equal weighting on each person).

Would be curious how this would play out for LLMs too. What do people actually want? I would guess most people think OpenAI is being too restrictive on what ChatGPT does, but would be curious to actually see that play out and how people are thinking about it.