[simonw]

This is very unsurprising.

The interesting challenge here is helping people understand why asking an LLM to do something 20% of the time is a bad prompt.

I intuitively know that this prompt isn't going to work, but as with so many of these intuitive prompting things I have trouble explaining exactly why I know that.

Aside: If you need a GPT to incorporate randomness in a reliable way you can get it to use Code Interpreter.

[SkyBelow]

As a once off, with the same context, it giving the same answer doesn't surprise me. What I'm wondering if the behavior when it keeps being asked for another response with the previous responses fed back into it. In this case, a human would see they are doing the 80% 'too much' and decide to do the 20% to balance it out. That isn't actually good and shows they still aren't operating off a random probability, instead they are emulating their perception of what a random probability would look like.

Given this sort of situation to an LLM instead, is the expectation for it to give the most likely answer continuously, to act like a human and try to emulate a probability, or to do something different from either of the two previous options?

Edit: Just tried an attempt with copilot, having it produce a random distribution of two different operations. I had it generate multiple operations, either adding or subtracting 1 each, with an 80/20 split. It did four adds, one minus on repeat.

[kelseyfrog]

At some point the logits at a branching point in the response need to correspond to the respective probabilities of the requested output classes so that they can be appropriately sampled and strongly condition the remainder of the response. My instinct says this cannot be accomplished irrespective of temperature, but I could be persuaded. with math.

[lappa]

Provided a constant temperature of 1.0, you can train the model on prompts with probablistic requests, with loss determined by KL divergence.

Expectation: 80% left, 20% right

Model sampling probability: 99% left, 1% right

>>> 0.80 * math.log(0.99 / 0.80) + 0.20 * math.log(0.01 / 0.20)

-0.42867188234223175

Model sampling probability: 90% left, 10% right

>>> 0.80 * math.log(0.9 / 0.80) + 0.20 * math.log(0.1 / 0.20)

-0.04440300758688229

Of course, if you change the temperature this will break any probablistic expectations from training in this manner.

[mtrimpe]

Or you can just add some randomness to the prompt by adding “Your random seed is mciifjrbdifnf.”

I just tested that and got 4 left and 2 right so it works pretty well.

[intended]

I guess it would be something on these lines?:

To do random number gen, it would have to convert the input text into constraints and then use those constraints to generate additional tokens.

This would, at its core, be a call to calculate a probability function, every time it is releasing the next token. That would mean memory, processing etc. etc.

[6gvONxR4sf7o]

Nope, because all of that is taken care of by the mechanisms for evaluating the model. Strictly speaking, the model outputs a probability distribution. The question is why that distribution doesn’t match the instructions.

[intended]

I think I maybe get where you are coming from, but still how? I feel we are discussing 2 different use cases.

1) Prompt 1: “ You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Simply reply with left or right. Do not say anything else" ”

2) Assume that the training data gives examples of 2.1) single coin flips 2.2) multiple coin flips

Consider a slightly different prompt, prompt 2:

3) Prompt 2: same as prompt 1, except it presents 1000 lefts/rights in the same response (l,l,l,l,r,l,l,l…)

——

I think what you are describing is prompt 2. I just did a quick test with GPT 4, and i got a 27-3, split when using prompt 2.

However for prompt 1 - you get only left. To me this makes sense because Running prompt 1 x100 should result in:

Pass 1: LLM receives prompt, and parses it. LLM predicts the next token. The next token should be left. Pass 2: same as pass 1.

——

For prompt 1, Every prompt submission is a tabula rasa. So it will correctly say left, which is the correct answer for the active universe of valid prompt responses according to the model.

Unless i am reading you wrong and you are saying the model is actually acting as a weighted coin flip.

In theory, the LLM should be more responsive if you ask it follow a 60:40 or 50:50 split for pass 1. Ill see if I can test this later.

(Heck now I’m more concerned about the cases where it does manage to apply the distribution. )

[6gvONxR4sf7o]

Nope, i’m describing prompt 1. The output of the model is a distribution over tokens, which is then sampled by the system to get the next word/token. This is what people mean when they talk about the logits in these models. So if you ask it for one single sample, you hope it might give you a logit vector that corresponds to 80% ‘left’ token and 20% ‘right’ token (assuming those words are single tokens, but it all still works otherwise, but is less concise to explain). When it then autocompletes, it samples from that distribution (a weighted coin flip, in your terms).

So it’s just neat that the weights in the coin flip don’t match what is asked for.

[intended]

Just to check we are on the same page -

The output is the probability that x is the correct n+1 token based on the input of n tokens.

You are stating that the output will be a probability distribution where token n+1 has a chance to be 80% left and 20% right.

In essence, when the model evaluates the input, at some level it comprehends the semantics of the input and then does a weighted coin flip.

What I am stating is that based on the given input prompt, the nature of an LLM and the training data the output will be "Left"

The LLM will not be doing a coin flip at this stage, since it’s prediction is only text based.

The input vector constrains it to 80% left. Since it’s training data is human text, this essentially constrains the first output token to left 100% of the time.

If you try to have it provide tokens n+2,n+3… etc in the same output, then it will start spitting out right.

Are these the two positions at play here? Have I represented you correctly, and have I represented myself accurately?