[cpitman]

I was looking for a more direct measure of this, how often a model "leaked" private state into public state. In a game like this you probably want to sometimes share secrets, but if it happens constantly I would suspect the model struggles to differentiate.

I occasionally try to ask a model to tell a story and give it a hidden motivation of a character, and so far the results are almost always the model just straight out saying the secret.

[Gracana]

Yup, that's the problem I run into. You give it some lore to draw on or describe a character or give them some knowledge, and it'll just blurt it out when it finds a place for it. It takes a lot of prompting to get it to stop, and I haven't found a consistent method that works across models (or even across sessions).