[8n4vidtmkvmk]

I don't think it can give you story beats you shouldn't be aware of yet. Those beats wouldn't be fed into the prompt until the event happens. LLM can't spit out what it doesn't know.

It might indeed fail to reveal something it should but even that i think is unlikely if the harness steers it hard enough.

I think it could be fun. If you're always given 4 choices of what you can ask the NPC then your choices can be too obvious. If its open ended then you have to think a little what to say and ask.

[nkrisc]

Why couldn’t an LLM accidentally spit out story beats if they plausibly follow from the context and the player’s input into the conversation?

[hdjrudni]

What "context"? If it's coming from the player's input, then the LLM isn't spoiling anything.

e.g. let's say an important character dies in your game but that hasn't happened yet in this point in the story. You're talking to some NPC. That NPC can't possibly spoil that that character is going to die because the developer wouldn't feed that in as context. The NPC doesn't "know" the character is going to die. You, as the player, might be able to convince the NPC that that will happen and it might respond accordingly, but it won't just randomly divulge that by accident.

[td2]

Or worse, what if the story of the game was already part of the training data

Like a remake of a game

[hdjrudni]

This one is a fair point... if all the characters have distinctive names then NPC can plausibly become aware that it is a character in that game and spoil things. I don't know how to stop that, short of training a clean model from scratch, but that seems a bit much for a game.