[zahlman]

> Saying "I know how LLMs work because I read a paper about transformer architecture" is about as delusional as saying "I read a paper about transistors, and now I understand how Ryzen 9800X3D works". Maybe more so.

Which is to say, not delusional at all.

Or else we have to accept that basically hardly anyone "understands" anything. You set an unrealistic standard.

Beginners play abstract board games terribly. We don't say that this means they "don't understand" the game until they become experts; nor do we say that the experts "haven't understood" the game because it isn't strongly solved. Knowing the rules, consistently making legal moves and perhaps having some basic tactical ideas is generally considered sufficient.

Similarly, people who took the SICP course and didn't emerge thoroughly confused can reasonably be said to "understand how to program". They don't have to create MLOC-sized systems to prove it.

> It takes actual reverse engineering work to figure out how LLMs can do small bits and tiny slivers of what they do. And here you are - claiming that we actually already know everything there is to know about them.

No; it's a dismissal of the relevance of doing more detailed analysis, specifically to the question of what "understanding" entails.

The fact that a large pile of "transformers" is capable of producing the results we see now, may be surprising; and we may lack the mental resources needed to trace through a given calculation and ascribe aspects of the result to specific outputs from specific parts of the computation. But that just means it's a massive computation. It doesn't fundamentally change how that computation works, and doesn't negate the "understanding" thereof.

[ACCount37]

Understanding a transistor is an incredibly small part of how Ryzen 9800X3D does what it does.

Is it a foundational part? Yes. But if you have it and nothing else, that adds up to knowing almost nothing about how the whole CPU works. And you could come to understand much more than that without ever learning what a "transistor" even is.

Understanding low level foundations does not automatically confer the understanding of high level behaviors! I wish I could make THAT into a nail, and drive it into people's skulls, because I keep seeing people who INSIST on making this mistake over and over and over and over and over again.

[zahlman]

My entire point here is that one can, in fact, reasonably claim to "understand" a system without being able to model its high level behaviors. It's not a mistake; it's disagreeing with you about what the word "understand" means.

[Chabsff]

For the sake of this conversation "understanding" implicitly means "understand enough about it to be unimpressed".

This is what's being challenged: That you can discount LLMs as uninteresting because they are "just" probalistic inference machines. This completely underestimates just how far you can push the concept.

Your pedantic definition of understand might be technically correct. But that's not what's being discussed.

That is, unless you assign metaphysical properties to the notion of intelligence. But the current consensus is that intelligence can be simulated, at least in principle.

[LeroyRaz]

I'm not sure what you mean?

Saying we understand the training process of LLMs does not mean that LLMs are not super impressive. They are shining testiments to the power of statistical modelling / machine learning. Arbitrarily reclassifying them as something else is not useful. It is simply untrue.

There is nothing wrong with being impressed by statistics... You seem to be saying that statistics is interesting and there for to say that LLMs are statistics dismissed them. I think perhaps you are just implicitly biased against statistics! :p

[knollimar]

Is understanding a system not implicitly saying you know how, on a high level, it works?

You'd have to know a lot about transformer architecture and some reasonable LLM specific stuff to do this beyond just those basics listed earlier.

When it's not just a black box but you can say something meaningful to approximate its high level behavior is where I'd put understand. Transistors won't get you to CPU archiecture and transformers don't get you to LLMs.

[whatevertrevor]

There is so much complexity in interactions of systems that is easy to miss.

Saying that one can understand a modern CPU by understanding how a transistor works is kinda akin to saying you can understand the operation of a country by understanding a human from it. It's a necessary step, probably, but definitely not sufficient.

It also reminds me of a pet peeve in software development where it's tempting to think you understand the system from the unit tests of each component, while all the interesting stuff happens when different components interact with each other in novel ways.