[orbital-decay]

This is a known and solved problem. Such a test is pointless for a general-purpose model, because like most people you're using multiturn chats in a naive way, fighting the default finetuning that is done intentionally.

1. You're sending your in-character inputs to an instruction-tuned model under the user role, in a multiturn chat. It's biased to treat these inputs as instructions and this behavior will show itself no matter what. Besides, the rigid structure of the assistant persona reply (usually tl;dr - explanation - "would you like to know more") is going to leak into such roleplay no matter what. To solve this problem on a generalist model you need to make your harness lump up all character turns into a seamless stream with formalized inline markers (e.g. screenplay-like paragraph prefixes or XML), use one of them as a custom stop string, send all this under one role (e.g. assistant), and prefill the assistant reply with a few messages from the past roleplay. This will break the rigid instruction-tuning structure (and also the cache, since caching breakpoints are based on chat turn boundaries in most APIs).

2. The models are simply not trained to "take incorrect actions back" in a story, this wouldn't make any sense. What happened is considered happened. This is a job for your harness, unless you want to make a specific finetune with a rigid format. You have to design and prompt it around the possibility of out-of-character user inputs, and think about how much freedom you want to give the user and how exactly you want to correct their actions. Validation with a second agent suggested in sibling comments is pretty good for this.

[katzenversteher]

You have some very good points and my approach was indeed naive. However, I believe it also shows that "general-purpose" models are not really general purpose and can't really step out of their assistant role. A common and often promoted prompting technique is to prompt a model to "behave like ..." or "you are a ..." which means these instructions do not really work.

[moritzwarhier]

> However, I believe it also shows that "general-purpose" models are not really general purpose and can't really step out of their assistant role. A common and often promoted prompting technique is to prompt a model to "behave like ..." or "you are a ..." which means these instructions do not really work.

I think you're still confusing model and "LLM app" there.

I'm not that versed myself in these things, but you can, for example, look at the conversation templates, stop markers etc. in open weight models on HuggingFace, or play around with these things by yourself and modify them using llama.cpp or ollama (the things I mention in this paragraph are, AFAIK, not part of the model). These, and parameters like temperature, sampling etc. are just the things that can be controlled without touching the model.

Of course, frontier models and their uses have supposably a lot more machinery built around them to orchestrate their usage, apart from even chatbots defaulting to "agentic" behavior for many use cases.

And models still are specialized, and fine-tuned for instruction usage, so things like the conversation template, system prompts won't be enough to bend the characteristics of such a model in all desired directions. But "general-purpose model" has become a very fuzzy term by now.

[orbital-decay]

The model is trained to follow the default template pretty closely, breaking it usually results in much worse performance and better output variance. Certain models with synthetic data in pre-training can melt down completely. At some point into this breakage you can just take the base model and it will be better.

If you want to use a custom chat scheme, use it as an overlay, don't break the default chat/tool use/reasoning template.

[moritzwarhier]

Thanks!

Matches my superficial experiments with trying to tweak Ollama's "modelfile" using some LLaMa- or gpt-oss-based instruction-tuned model as "base".

I need to experiment more with base models. The time from the end of 2019 onwards, when I first came across talktotransformer, it felt so magical.

Getting meaningful things out of these things can feel so... restraining.

And on the other hand: I'm tbh freshly stuck in the stage of being amazed at what current frontier coding models and apps can do.

[dvfjsdhgfv]

> This will break the rigid instruction-tuning structure

Well, this is something one might naively hope for, unfortunately it only works to a certain extent.

[orbital-decay]

Can you elaborate? Where does the structure come from in this case? The model can't even see the boundaries of the reply (it's started with a prefill and stopped with a custom stopping string).

[charcircuit]

This is like saying coding is a known and solved problem and benchmarks for coding is pointless for a general model. And that using a model for coding is an abuse of the multiturn chats AIs are tuned for.

1. The user should be able to prompt the AI to act differently from its default behavior. A human assistant is capable of role playing without always sounding like an assistant.

2. If the user asks the AI to follow the script and not allow unrealistic things to happen it should push back. The user is not always absolutely correct.

[orbital-decay]

You have a point, this might help test the resistance to jailbreaks for example. There are probably better tests for that than OOC roleplaying, I think. At the same time, what you expect is likely not going to happen due to many reasons (e.g. can't optimize for two contradicting objectives), and harness design offers a practical workaround that is being used for years already and is reliable unless the user is actively trying to jailbreak the model+harness.

[pegasus]

Prefill isn't supported in the major commercial models anymore. Probably in order to thwart jailbreak attacks.

[orbital-decay]

Only on GPT (never was) and more recently Claude, and it still can be simulated with structured outputs for the purpose above.

Even if it's not supported somewhere (e.g. z.ai API which isn't mature enough and has neither assistant prefills nor actual structured outputs), it's still better and more seamless than using the default user/assistant scaffolding for role alternation.

[ryoshu]

Are you testing a model or a harness? People conflate the two.