[Kiro]

The way I understand these models (and please correct me if I'm wrong) is that they predict every single word one at a time out of billions and billions of possible paths. So for the model to actually reproduce code you would either need to bait it really hard (reducing the possible paths by pushing it into a corner) or be impossibly "lucky". It can't just copy-paste something accidentally since it has no "awarenesses" of the source material the last neuron was created from.

[Jensson]

It is trivial to encode text in neural networks, these models try hard to avoid that by punishing it in training but they still encode a lot of text word for word. The most famous example is "Fast inverse square root", it gives you the exact same thing with same comments etc.

[Kiro]

When you say it's trivial to encode text in neural networks, what does that mean for LLMs? What makes it decide to encode certain texts or not? Isn't it just one big network of neurons?

The prompt I've seen for it to verbatim reproduce the fast inverse square root from Quake was:

    // fast inverse square root

    float Q_

When I ask ChatGPT to give me code for a fast inverse square root it doesn't reproduce it at all but gives me an implementation that looks completely different.

So, my original thought was that the prompt above with the characteristic Quake III Q_ naming is enough to push it into a corner where the path is reduced to just one possibility (with that path being the words in the code itself) and not that it merely copypasted the code from an encoded version of it. I.e. it still predicts it word-by-word but with only one possible way for each step. This is just be my naive take on it though but I really want to understand.

[Jensson]

> my original thought was that the prompt above with the characteristic Quake III Q_ naming is enough to push it into a corner where the path is reduced to just one possibility

This is what people mean when they say it copy pastes things. It doesn't literally go to the source code, press ctrl-c and then ctrl-v that to you, nobody believes it did.

And the model does this a lot, as I said the reason it doesn't do that all the time is that they train it not to. And the quake code example got such a big deal that they started to hard code it to not return that, but that doesn't mean it never does that for other things, just that this particular example is now "fixed".

[Kiro]

Alright, that makes sense. I still think it's an important distinction and just looking at this thread it certainly seems that people think ChatGPT is just copypasting things. Like if it ctrl-c/ctrl-v all this from a single tutorial on how to make a puzzle game.

The Quake code example still works in Copilot so I don't think they did anything about it, but only if I use the Q_ trick.

[ldhough]

I think they did something specific to this example to make it no longer generate the code verbatim. A few months ago when I tried it "fast inverse square root" was enough to get the exact function with comments and no explanation. A week ago that gave me the same function with no comments and

"give me C code for calculating the fast inverse square root of a number, do not explain yourself"

again resulted in the original function, comments included. Today it generates a completely different result and interestingly opts to explain the origin of the technique despite the directive for it to not explain itself.