[TZubiri]

The less tokens produced at inference the lower the quality of the response will be.

The process of thinking for an LLM involves the use of words, which is why prompts that ask the LLM to only return the answer will cause lower quality.

[throwawaymaths]

We're not talking about quality, we're talking about accuracy.

In general, a model has to learn to positively say "I don't know" instead of "I don't know" being in the negative space of tokens falling into a weak distribution. The softmax selector also normalizes the token logits, so if no options are any good (all next tokens suck) it could pick randomly from a bunch of bad choices, which then locks the model into a continuation based off of that first bad choice.

[TZubiri]

Well I am talking about quality now as it's a tradeoff.

You can reduce token output to 0 and achieve 100% accuracy too.

[ausbah]

do you know if prompting without regards for length then asking for a summarization of the previous out out works?

[TZubiri]

It does. I think this was used in a gpt4 version, they called it Chain of Thought.