[insanitybit]

No, it's actually executing that Python code. This is what allows an LLM (or an 'llm based system', I guess) to do something like "reverse <some uuid that has never been observed before>" - it can't just memorize the output and map it to the input because the output has literally never been observed. Instead, if it knows the algorithm for reversing a string, it can just use that and offload the execution to Python.

[runarberg]

That is even less impressive. I was thinking—like normal linear models—it would be capable of interpolation.

[throwup238]

It is. It doesn’t even need an existing language. You can define your own psuedo language in the prompt and have ChatGPT “execute” it (works best with 4 nonturbo).

You can even combine your pseudo language with natural language. See the OP’s custom GPT and the comments here: https://news.ycombinator.com/item?id=38594521

[insanitybit]

That looks totally different. In the case of the Python code it is literally executing it by calling out to CPython.

[no_identd]

…got a source for that claim?

[throwup238]

https://help.openai.com/en/articles/8302319-why-can-t-i-see-...

https://help.openai.com/en/articles/8437071-advanced-data-an...

[labaron]

I checked those links and didn’t see it mentioned that python code is actually executed. Could you quote the relevant part?

[insanitybit]

Why is it less impressive?

[runarberg]

I would say creating a model which is able to interpolate from training data in a way which produces an accurate output of a new input is a little impressive (if only as a neat party trick), however anybody can run a python interpreter on a server somewhere.

I’m sure there are use cases for this. But in the end it is only a simple feature added onto a—sometimes—marginally related service.

[insanitybit]

Hm, I don't think of it that way I guess. What the LLM is doing is generalizing a problem based on previous problems it has seen and then offloading the execution of that problem to a machine with some defined, specific semantics.

This is a lot more than a party trick. The model is able to describe the program it wants to execute and now it can accurately execute that - that it 'offloads' the work to a specialized program seems fine to me.

It's way more than a simple feature, this is enabling it to overcome one of the biggest limitations and criticisms of LLMs - it can answer questions it has never seen before.