[feoren]

You are assuming there is no such thing as emergent complexity. I would argue the opposite. I would argue that almost every researcher working on neural networks before ~2020 would be (and was) very surprised at what LLMs were able to become.

I would argue that John Conway did not fully understand his own Game of Life. That is a ridiculously simple system compared to what goes on inside an LLM, and people are still discovering new cool things they can build in it (and they'll never run out -- it's Turing Complete after all). It turns out those few rules allow infinite emergent complexity.

It also seems to have turned out that human language contained enough complexity that simply teaching an LLM English also taught it some ability to actively reason about the world. I find that surprising. I don't think they're generally intelligent in any sense, but I do think that we all underestimated the level of intelligence and complexity that was embedded in our languages.

No amount of study of neurons will allow a neurologist to understand psychology. Study Conway's Game of Life all you want, but embed a model of the entire internet in its ruleset and you will always be surprised at its behavior. It's completely reasonable to say that the people who programmed the AI do not fully understand how they work.

[Azkron]

Whatever comes out of any LLM will directly depend on the data you feed it and which answers you reinforce as correct. There is nothing unknown or mystical about it. I honestly think that the main reason big tech claims they “don’t understand how they work” is either to avoid responsibility for what comes out of them or as a marketing strategy to impress the public.

EDIT: By the way, I definitely think LLMs are intelligent and could even be considered “synthetic minds.” That’s not to say they are sentient, but they will definitely be subject to all kinds of psychological phenomena, which is very interesting. However, this is outside the scope of my initial comment.

[feoren]

> Whatever comes out of any LLM will directly depend on the data you feed it

Right, and whatever comes out of Conway's Game of Life will directly depend on its initial setup as well. Show me a configuration of Conway's Game of Life that is tailored to emulate human speech and trained on the entire internet and then tell me your prediction of how it will evolve. You will get it completely wrong. Emergent behavior is a real thing.

> There is nothing unknown or mystical about it.

Almost all researchers and practitioners in the field seem to disagree with you on this. It is surprising that teaching a system to be extremely good at auto-completing English text is enough for it to develop an ability to reason. I happen to believe that this is more of an emergent property of our language than of neural networks, but it was definitely not predicted by almost anyone, not easily explainable, and maybe even a bit mystical-feeling.

Ph.D. dissertations have been published about trying to understand what is happening inside large neural networks. It's not as simple and obvious as you make it out to be.