[Azkron]

| "Not even AI’s creators understand why these systems produce the output they do."

I am so tired of this "NoBody kNows hoW LLMs WoRk". It fucking software. Sophisticated probability tables with self correction. Not magic. Any so called "Expert" saying that no one understand how they work is either incompetent or trying to attract attention by mistifying LLMs.

[feoren]

You are assuming there is no such thing as emergent complexity. I would argue the opposite. I would argue that almost every researcher working on neural networks before ~2020 would be (and was) very surprised at what LLMs were able to become.

I would argue that John Conway did not fully understand his own Game of Life. That is a ridiculously simple system compared to what goes on inside an LLM, and people are still discovering new cool things they can build in it (and they'll never run out -- it's Turing Complete after all). It turns out those few rules allow infinite emergent complexity.

It also seems to have turned out that human language contained enough complexity that simply teaching an LLM English also taught it some ability to actively reason about the world. I find that surprising. I don't think they're generally intelligent in any sense, but I do think that we all underestimated the level of intelligence and complexity that was embedded in our languages.

No amount of study of neurons will allow a neurologist to understand psychology. Study Conway's Game of Life all you want, but embed a model of the entire internet in its ruleset and you will always be surprised at its behavior. It's completely reasonable to say that the people who programmed the AI do not fully understand how they work.

[Azkron]

Whatever comes out of any LLM will directly depend on the data you feed it and which answers you reinforce as correct. There is nothing unknown or mystical about it. I honestly think that the main reason big tech claims they “don’t understand how they work” is either to avoid responsibility for what comes out of them or as a marketing strategy to impress the public.

EDIT: By the way, I definitely think LLMs are intelligent and could even be considered “synthetic minds.” That’s not to say they are sentient, but they will definitely be subject to all kinds of psychological phenomena, which is very interesting. However, this is outside the scope of my initial comment.

[feoren]

> Whatever comes out of any LLM will directly depend on the data you feed it

Right, and whatever comes out of Conway's Game of Life will directly depend on its initial setup as well. Show me a configuration of Conway's Game of Life that is tailored to emulate human speech and trained on the entire internet and then tell me your prediction of how it will evolve. You will get it completely wrong. Emergent behavior is a real thing.

> There is nothing unknown or mystical about it.

Almost all researchers and practitioners in the field seem to disagree with you on this. It is surprising that teaching a system to be extremely good at auto-completing English text is enough for it to develop an ability to reason. I happen to believe that this is more of an emergent property of our language than of neural networks, but it was definitely not predicted by almost anyone, not easily explainable, and maybe even a bit mystical-feeling.

Ph.D. dissertations have been published about trying to understand what is happening inside large neural networks. It's not as simple and obvious as you make it out to be.

[lappa]

This isn't suggesting no one understands how these models are architected, nor is anyone saying that SDPA / matrix multiplication isn't understood by those who create these systems.

What's being said is that the result of training and the way in which information is processed in latent space is opaque.

There are strategies to dissect a models inner workings, but this is an active field of research and incomplete.

[Azkron]

Whatever comes out of any LLM will directly depend upon the data you fed it and which answers your reinforced as correct. There is nothing unknown or mystical about it.

[richardatlarge]

The same could be said of people, revealing the emptiness of this idea. Knowing the process at a mechanism level says nothing about the outcome. Some people output German, some English. It’s sub-mechanisms are plastic and emergent

[wrs]

So many words there carrying too much weight. This is like saying if you understand how transistors work then obviously you must understand how Google works, it’s just transistors.

[Azkron]

I guarantee you that whoever designed Google understands how Google works.

[solarwindy]

The relevant research field is known as mechanistic interpretability. See:

https://arxiv.org/abs/2404.14082

https://www.anthropic.com/research/mapping-mind-language-mod...

[cma]

This is a bit like saying a computer engineer who wrote and understands a simple RISC machine in college thereby automatically understands all programs that could be compiled for it.

[Azkron]

No this is like saying that whovever writes a piece of software understands how it works. Unless one forgot about it or stumbled upon it out of sheer luck. And neither of those are the case with LLMs.