[capitol_]

> GPT3 was able to change its answers in believable ways depending on what demographic I instructed it to be.

Doesn't this just mean that your own preconceptions about those demographics matches the language models preconceptions? How would we know that is matches reality when presented with novel ideas/concepts that we want to get feedback on?

[doctoboggan]

That's part of what I mean by it needing to be calibrated. Initially some polling could be done with real people and the GPT agents. Whatever calibration factors are needed to make those two line up could then be used when asking the GPT agents novel questions.