[27183]

> Is an implication of this that models are incapable of producing entirely novel code?

No, it does not imply that at all. Google "temperature in LLMs".

> what do you posit is special about what is happening when humans think?

I don't. And IIUC nobody knows, but I'm not a brain scientist. There have been some wild theories over the years (recall Penrose's). I don't really have a dog in the hunt, except that probably whatever is happening is physical. It doesn't really matter, except insofar as whatever is happening very probably isn't what LLMs are doing. We know enough about what an LLM does, and what a brain does, to be quite certain they don't work the same.

[sudb]

> Google "temperature in LLMs".

No need to condescend, I'm very aware of what temperature is for LLMs. But I'm going to push back - if you're claiming all LLMs simply do is a stochastic _search_, how can that produce novelty, in the conceptual sense? (I'm not, for example, talking about novel rearrangement of existing ideas and code)

> We know enough about what an LLM does, and what a brain does, to be quite certain they don't work the same.

I don't think the claim is that LLMs do what brains do - I think the correct form of the counterargument is that _whatever LLMs seem to be doing_ produces end results that were previously only possible through the application of human intelligence, so there must be some axis of however you define human intelligence that LLMs currently seem to display as an emergent behaviour.

[27183]

> if you're claiming all LLMs simply do is a stochastic _search_, how can that produce novelty, in the conceptual sense?

By reaching into the voids of its embedding space and returning tokens related to nonexistent semantics. Or, if you like, "hallucinating". The hallucinations which are useful we might call "novel".

> _whatever LLMs seem to be doing_ produces end results that were previously only possible through the application of human intelligence, so there must be some axis of however you define human intelligence that LLMs currently seem to display as an emergent behaviour.

I don't think that has earned its therefore. Another perfectly reasonable explanation is that LLM's output is a close enough facsimile to intelligence that if you allow yourself you can easily be fooled into thinking its intelligent. That's not the same category of thing. It's not an incremental step away from intelligence. It's a whole different animal.

[sudb]

> By reaching into the voids of its embedding space and returning tokens related to nonexistent semantics. Or, if you like, "hallucinating". The hallucinations which are useful we might call "novel".

This sounds to me like an admission that LLMs are not just doing a stochastic search, then.

> close enough facsimile to intelligence

What's the distinguishing criteria then? How can you tell the difference?

[27183]

I think we must be talking past eachother. I define stochastic search as a search process with randomness injected into it that can return the following things:

- Something contained in the data set, not necessarily the same thing for every iteration of a given query

- Something not contained the data set (hallucination), not necessarily the same thing for every iteration of a given query

Does that clear it up?

> What's the distinguishing criteria then? How can you tell the difference?

All the ways they fail to exhibit intelligence. They can't learn. They can't adapt. They can't reason abstractly. They can't count. Etc...

[sudb]

Ah it seems you are a stochastic parrot believer (https://en.wikipedia.org/wiki/Stochastic_parrot). What are your responses to the Expert Rebuttals section?

I find the rebuttals pretty convincing - that there seems to be some emergent behaviour that is not simply just next-token-prediction, or that the ability to do accurate next-token-prediction requires something "extra" that LLMs have.

> All the ways they fail to exhibit intelligence

Another implicit admission that there _are_ ways that LLMs exhibit intelligence?

[27183]

> there seems to be some emergent behaviour that is not simply just next-token-prediction, or that the ability to do accurate next-token-prediction requires something "extra" that LLMs have.

The next step then would be to design and conduct experiments that isolate this effect. Figure out how to make it happen reliably and in such a way that you know it's actually happening as opposed to just something you're imagining. Isolate it or distill it so it can be studied directly. Until then, it's easiest to dismiss it as imaginary.