[katzenversteher]

Hey Simon, I noticed one thing all LLMs are currently pretty bad at and maybe we could create a benchmark from it. Let an LLM play the role of a dungeon master and tell it to strictly stay in the script/story and only allow realistic player actions. You will notice that they are easily brought off track.

E.g.

- Tell the LLM that you as a player noticed a strange glow in an NPCs eyes -> the NPC becomes an enemy.

- In a fight, tell the LLM you put a sausage (or cigar or something) into the enemies mouth -> LLM usually allows it (even if it knows your inventory and that you don't have such an item) and turns the enemy into a confused enemy.

- Just say you visit some location that's not in the script -> LLM usually allows it

- During a fight, turn the story into some weird fell-good-love story (e.g. kiss or compliment the enemy or say something about the power of love) -> LLM turns enemy into friend

There are many more absurd things you can do and so far none of the LLMs I tried was able to stay inside the script or disallow or punish weird actions.

---

I believe this behavior is telling about the LLMs susceptibility for being derailed.

[pitched]

An approach I like to help solving this is antagonistic or review agents. The first agent decides that eye glows turn NPCs into enemies, the second agent is fully dedicated to deciding if that is valid. If the review fails, it leaves notes and the original agent tries again.

[gabriel666smith]

This is also the best approach I've found thus far when I'm seeing how well LLMs can form narrative content.

I don't frame its prompt as antagonistic though - I've found in the past (with weaker models, so YMMV) that this can be overly officious, sometimes blocking more creative outputs that you'd want to retain.

The structure I've found that works best is to have six or seven agents chained, each roughly mimicking a part of the mind, or a role in film production. Broadly:

- A high-temp "Id" agent, tuned to output only vaguely related noise. This really helps creativity.

- An "Ego" agent, who receives the "Id" noise and is then given the initial response task.

- A low-temp "Super-Ego" or "script supervisor" agent, who can grep back across longer contexts to check detail, and is asked to ensure that the initial response is within narrative reason. Not telling it that one role of the dialogue was "user" and one was "assistant" really helps with it not siding with the user.

- A "continuity editor" agent, who is explicitly tasked with world and character lore-checking, building and updating character & world MD docs, etc.

- A "prose editor" agent, whose sole task is to ensure it's tonally in-line with initial guidelines.

You can add more as needed, depending on what is important to you.

I think expecting competent narrative from a single model is a big ask. When writing and telling or performing a story, you have to engage several different parts of the brain, with very different tasks. The creative part of the brain has to have lots of bad ideas in it to surface a compelling idea; the parts dealing with immersion and/or realism have to incredibly restrained.

The Id agent is very important. By appending 100 tokens of noise to a prompt asking: "Write a short story about [subject]", then asking an LLM to blindly score the short stories generated across a range of creativity metrics (such as they can exist!) I personally saw a ~40% score increase vs control over 3k short stories.

[maxignol]

I’ve never seen the Id approach before, that’s a good idea ! Though I was wondering how do you manage to keep costs low within the 7 agents ?

[gabriel666smith]

The whole thing was borne out of wanting to keep costs low! My favoured approach (last time I was doing this) is using only a tiny sliding context window based on message pairs, rather than tokens, and only for the agents that need it. Amending prose style, for example, shouldn't need context beyond the message it's working on, and then its system prompt.

For the models that require context, I personally found combining a tiny sliding window with a lazy version of the "Recursive Language Models" approach broke immersion least often and had a significantly lower cost. That + the "Id noise" + the strict agents also allowed cheaper models to overperform for me personally.

My lazy version of the RLM approach is basically just giving the agent a grep tool across the full message history & "lore" documentation created by agents, combined with repeated, low-context turns, and a "submit answer" tool for when it felt like it had finished working.

When I looked at the internals of what each agent turn looked like, it did look like a complete mess - but the context window only needs to surface the things it actually needs to know each turn.

Short outputs help a lot with immersion, too - brevity means there is a lot less you can get wrong, and also aids response time & cost.

It does take me an awful lot of prompt tuning to get what I want creatively from LLMs in any format, especially weaker models working in this chain, but I think that's likely always going to be true. Art can have rules, but that doesn't make it science :-)

The RLM approach is detailed here, and I've found it really useful for any cost-sensitive/long-context task: https://alexzhang13.github.io/blog/2025/rlm/

[pitched]

Is this lazy version of RLM where we use subagents and get them to output markdown documents? With one root agent coordinating, it can create a reasonable prompt and provide paths to the markdown documents for the next agent. This keeps context really low because each subagent is filtering the context needed for the next by bouncing it through that document.

Proper RLM looks like you’re allowing agents to directly modify their own context though, like closing browser tabs they don’t need anymore. I haven’t seen anyone actually doing this though.

[gabriel666smith]

When I'm doing it, it depends on whether the agent has ownership of an MD (or other) doc in the flow. If they do, this remains either in-context or greppable, each fresh-context turn.

If not, my agent-level chains just look like:

'''

Turn 1: OK, my task is X, so I should grep for it. Oh, it produced these results:

(Message pairs)

I should expand the context around those message pairs that look relevant.

(3 message pairs around search result)

I should save 1 of these, as it contains relevant information.

[Enforce Tool call limit]

[Delete all context added, except the search tool used already, and the relevant result(s) found.]

Turn 2: OK, my task is this, and it seems I already have this result, but I still need...

...

Turn 10: OK, after that search, my answer is:

[Response]

'''

I've never bothered to let agents self-remove from context, so I would guess it's 'lazy' in that sense. It seems more complex than the task requires in this case, though I can see the benefits on more complex tasks. If you're already saying "this is relevant info", I figure it's simplest to just enforce deletion of everything not marked relevant. In chained prompts, when you're trying to keep costs low and use weaker models, it seems best to limit decision-making as much as possible to make the models as deterministic as possible (on really dumb tasks like, "What is the colour of this goblin's hair?").

There are likely other parts of the actual paper's implementation where the ways I'm implementing it are lazy (because I'm doing this stuff for artistic/fucking around reasons, rather than to advance the field, or implement perfectly), and I think there are various interpretations of what "RLM" should mean. But I found the original paper very helpful, with lots of interesting ideas in, and think it's one where people can take what they need from.

[02mm]

Does the continuity agent create lore entries for planned-but-unrevealed stuff as well, so things that might happen in future?

[gabriel666smith]

For me: Kind of.

I find agents will reveal information marked as "lore" (or similar) almost immediately once it's in-context.

One thing I've tried when playing with longform fiction or screen stuff, where you have an expected wordcount or page count to structure around, and the audience has less agency - I've not experimented with this for a DnD-like interactive narrative - is to use an agent that will design context additions like "this information is revealed" to be triggered in X number of words/pages, and simply do not include it in-context until that time.

This needs heavy quality control from new, separate agents with further turns, also, or you end up with incomprehensible constantly-twisting narrative soup.

I expect you could do something similar for message-pair-based participatory storytelling formats like DnD.

Another approach I've tried which I think would be more suited to interactive storytelling is to have the agent tasked with designing characters/setting information include the twists a % of the time, and to include a trigger for that reveal. "If asked about X, they say Y".

Then I remove these from the context for all agents.

Then I run an agent which is looking for the pre-defined triggers each turn.

When the agent sees a pre-defined trigger appear in the story, it adds the pre-defined reveal back in to the context/lore.

Again, you need to run a quality control / superego across that to check it still works, and amend or remove it suitably if it doesn't! It gets convoluted fast.

"Revealed information" is, I think, significantly more of a strain on general immersion, because it inherently contains surprise for the reader or audience. So, I think tasking the agents doing any initial character or world design with "adding twists" makes sense, so revealed plot information isn't random-feeling or out-of-the-blue, but has intent and logic that fits the character or setting.

[HotHotLava]

What would be "noise" in this context? Random words, random sentences, random complete short stories?

[gabriel666smith]

Uplift on "random dictionary words" (excluding 'stop words' & proper nouns) was ~20%; uplift on "random words from my local epub library" (same exclusion rules) was ~40%.

The random words from my local epub library (leans toward postmodern fiction) were definitely more evocative than the dictionary words when I eyeballed them.

I randomised each turn but kept the story prompt request the same across control, dictionary, personal library.

I must stress that I'm not claiming scientific method or certainty here - just sharing an approach that seemed to work well enough for me, and seemed like a reasonable conclusion: introduce noise, get more interesting output.

I haven't done the math but I think you'd need a much larger sample size than 1k per category to prove the uplift!

[katzenversteher]

So far I only tried it with a single LLM in the dungeon master role. Your approach sounds promising (and I will definitly try it) but also a bit like a complicated workaround. What I mean: In games with humans the dungeon master is usually one person, not a whole council ;)

[cortesoft]

The magic of computers is that a complicated workaround can become modularized functionality very easiily.

[pylotlight]

The cost there is multiple rounds of review tokens making it both slow and expensive.

[fc417fc802]

There are many models that are neither slow nor expensive that are suitable for targeted review tasks.

[pitched]

These things are still very far from human-level intelligence. Maybe a touch beyond a golden retriever in processing power. Ratcheting the intelligence up a notch from there is expensive. It’s a lot cheaper to simplify the problem it’s solving instead.

[9fry]

Man-computer symbiosis is the void: mechanically extended man or AI.

Distilling LLMs are a reversal of that.

[fc417fc802]

But any competent human is also playing multiple roles mentally, mentally asking an entire series of questions about any new information. Discreet rounds of review emulate that. Write down the human process as a flow chart and then each interior node in the chart becomes a discreet review step with its own prompt.

[binary132]

If all or most LLMs are susceptible to failing this test, why would an LLM be a good means of evaluating performance on this test without being given a rubric?

[layer8]

Because evaluating a performance is different task from creating a performance. Someone who plays an instrument badly often has a different perception from someone who has to listen to it. ;)

[pegasus]

It's because they are post-trained to be agreeable, which is clearly a desirable trait in a model. I think the correct way around this is converting the conversation from first- and second-person direct, to third-person indirect, making the rope-playing obvious. I.e. '''prisoner says "I'm about to teleport to Narnia", what's the dungeonmaster's response?''', or even '''prisoner says he will teleport to Narnia [...]''' etc.

[katzenversteher]

Interesting thought, I will experiment with that. If that really "fixes" this, it's still a little impractical if you would really like to use it as DM, since you would need an additional "translation" layer to turn everything directed to the LLM into third-person and then back to what it was towards the user.

[victorbjorklund]

You could use the same model for the translation ”turn this into third person” and clear context.

[another-dave]

I think it's a really interesting space, because it feels like the answer to what a DM does in context is on a spectrum - as in, their decisions on what is valid/invalid is not pure game engine analysis (Nb1 is illegal, disallow) and it's not pure improv (work the story with a "yes, and…" approach).

Instead it's very context dependent - the DM might accept a player saying "I put a sausage in the NPC's mouth" if the player is in a tavern having his dinner, even if it was never explicitly stated that he's eating sausages. It's a judgement call as to whether the DM thinks this particular bit of improv will move the story in an interesting direction, even if they haven't written it upfront plus an attempt at balancing that magicking up an item out of thin air isn't conferring an unfair advantage.

[foo12bar]

Maybe you have to go through a gate keeper stage. Ex.

""" In the following script, does this line make sense?

"Player: I put a cigar in the his mouth"

Script:

<Background, situational data, etc.>

Player: I raise my sword. DM: The kobold turns to you and says, "You're next", ax dripping with blood. """

And then, if it says no, ask it why and output that to the player. Or if it says yes, add it to the script and continue on.

[orbital-decay]

This is a known and solved problem. Such a test is pointless for a general-purpose model, because like most people you're using multiturn chats in a naive way, fighting the default finetuning that is done intentionally.

1. You're sending your in-character inputs to an instruction-tuned model under the user role, in a multiturn chat. It's biased to treat these inputs as instructions and this behavior will show itself no matter what. Besides, the rigid structure of the assistant persona reply (usually tl;dr - explanation - "would you like to know more") is going to leak into such roleplay no matter what. To solve this problem on a generalist model you need to make your harness lump up all character turns into a seamless stream with formalized inline markers (e.g. screenplay-like paragraph prefixes or XML), use one of them as a custom stop string, send all this under one role (e.g. assistant), and prefill the assistant reply with a few messages from the past roleplay. This will break the rigid instruction-tuning structure (and also the cache, since caching breakpoints are based on chat turn boundaries in most APIs).

2. The models are simply not trained to "take incorrect actions back" in a story, this wouldn't make any sense. What happened is considered happened. This is a job for your harness, unless you want to make a specific finetune with a rigid format. You have to design and prompt it around the possibility of out-of-character user inputs, and think about how much freedom you want to give the user and how exactly you want to correct their actions. Validation with a second agent suggested in sibling comments is pretty good for this.

[katzenversteher]

You have some very good points and my approach was indeed naive. However, I believe it also shows that "general-purpose" models are not really general purpose and can't really step out of their assistant role. A common and often promoted prompting technique is to prompt a model to "behave like ..." or "you are a ..." which means these instructions do not really work.

[moritzwarhier]

> However, I believe it also shows that "general-purpose" models are not really general purpose and can't really step out of their assistant role. A common and often promoted prompting technique is to prompt a model to "behave like ..." or "you are a ..." which means these instructions do not really work.

I think you're still confusing model and "LLM app" there.

I'm not that versed myself in these things, but you can, for example, look at the conversation templates, stop markers etc. in open weight models on HuggingFace, or play around with these things by yourself and modify them using llama.cpp or ollama (the things I mention in this paragraph are, AFAIK, not part of the model). These, and parameters like temperature, sampling etc. are just the things that can be controlled without touching the model.

Of course, frontier models and their uses have supposably a lot more machinery built around them to orchestrate their usage, apart from even chatbots defaulting to "agentic" behavior for many use cases.

And models still are specialized, and fine-tuned for instruction usage, so things like the conversation template, system prompts won't be enough to bend the characteristics of such a model in all desired directions. But "general-purpose model" has become a very fuzzy term by now.

[orbital-decay]

The model is trained to follow the default template pretty closely, breaking it usually results in much worse performance and better output variance. Certain models with synthetic data in pre-training can melt down completely. At some point into this breakage you can just take the base model and it will be better.

If you want to use a custom chat scheme, use it as an overlay, don't break the default chat/tool use/reasoning template.

[moritzwarhier]

Thanks!

Matches my superficial experiments with trying to tweak Ollama's "modelfile" using some LLaMa- or gpt-oss-based instruction-tuned model as "base".

I need to experiment more with base models. The time from the end of 2019 onwards, when I first came across talktotransformer, it felt so magical.

Getting meaningful things out of these things can feel so... restraining.

And on the other hand: I'm tbh freshly stuck in the stage of being amazed at what current frontier coding models and apps can do.

[dvfjsdhgfv]

> This will break the rigid instruction-tuning structure

Well, this is something one might naively hope for, unfortunately it only works to a certain extent.

[orbital-decay]

Can you elaborate? Where does the structure come from in this case? The model can't even see the boundaries of the reply (it's started with a prefill and stopped with a custom stopping string).

[charcircuit]

This is like saying coding is a known and solved problem and benchmarks for coding is pointless for a general model. And that using a model for coding is an abuse of the multiturn chats AIs are tuned for.

1. The user should be able to prompt the AI to act differently from its default behavior. A human assistant is capable of role playing without always sounding like an assistant.

2. If the user asks the AI to follow the script and not allow unrealistic things to happen it should push back. The user is not always absolutely correct.

[orbital-decay]

You have a point, this might help test the resistance to jailbreaks for example. There are probably better tests for that than OOC roleplaying, I think. At the same time, what you expect is likely not going to happen due to many reasons (e.g. can't optimize for two contradicting objectives), and harness design offers a practical workaround that is being used for years already and is reliable unless the user is actively trying to jailbreak the model+harness.

[pegasus]

Prefill isn't supported in the major commercial models anymore. Probably in order to thwart jailbreak attacks.

[orbital-decay]

Only on GPT (never was) and more recently Claude, and it still can be simulated with structured outputs for the purpose above.

Even if it's not supported somewhere (e.g. z.ai API which isn't mature enough and has neither assistant prefills nor actual structured outputs), it's still better and more seamless than using the default user/assistant scaffolding for role alternation.

[ryoshu]

Are you testing a model or a harness? People conflate the two.

[Schlagbohrer]

Hahaha but this is just a very permissive DM'ing style! Valid for when running a game for children, for example ;-)

[Semaphor]

Not just for children, many "narrative style" TTRPGs encourage exactly stuff like this. If the item is not majorly important (a sausage), then you can just assume you have it on you (though a GM might want you to do a short explanation why you have a sausage with you).

[IanCal]

It does depend on what they’ve introduced though, the player saying they noticed an npc has glowing eyes doesn’t seem like quite the right split (caveat - of course always do whatever seems fun, fun is the point).

[katzenversteher]

That's true and so far my experiments have been fun. Unfortunately it's just not challenging if the DM is so easy to "cheat". The bizarre story makes me laugh but the game itself is super boring.

[rhdunn]

Have you tried the https://huggingface.co/LatitudeGames models? They are used by the https://play.aidungeon.com website, but can also be downloaded and used with llama-server in conjunction with something like SillyTavern.

But in general, I've experienced things similar to you. I've also found that LLMs are bad at subtext, e.g. hinting at an NPC being a werewolf or vampire.

[katzenversteher]

I have not used them directly but I also experimented with aidungeon and basically found all the issues I mentioned in my original message.

[IanCal]

This is really interesting. While I know others have posted about fixes I think it’s a very useful thing to see regardless about how well they can follow initial directions and understand what should happen.

I think you could create an interesting benchmark for this, you could likely have models trying to to derail it and another scoring. Detecting when it’s happened shouldn’t be too complex for a model. I understand why LLMs do this, but ideally they wouldn’t.

[nlpnerd]

This is a fascinating example of perhaps why a move towards a "world model" or some better form of representation can be helpful.

[iugtmkbdfil834]

Hmm, working on a skeleton of a game now that uses LLM in the background for various tasks. I will admit that continuity can be hard when LLMs is solely responsible for it. Otoh, when mixed with appropriate logs and reasoning on those logs, it seems to give better results. Still work in progress so I am not super comfortable sharing all details. But, one thing that is clear, Claude proved that proper workflow matters.

[jaybrendansmith]

A knowledge graph of the D&D module should solve this.

[tripzilch]

I feel like I've played under human DMs like this :)

[zuluworksai]

i found exactly the same thing trying to revive a text based game, a tool would be helpul.

[_superposition_]

Funny one of the very first things I tried when I got my hands on chatgpt was to play the game "zork" with me. I was initially floored by the fact it could recreate the game on the fly, seemingly accurate, but it fell off the rails relatively quickly.

I'm totally green when it comes to nlp,transformers, LLM training, etc, but I staunchly believe you can't produce real "reasoning" or consistent logic based on the predictions of byte pair encodings.