[latemedium]

I'm reminded of the metaphor that these models aren't constructed, they're "grown". It rings true in many ways - and in this context they're like organisms that must be studied using traditional scientific techniques that are more akin to biology than engineering.

[dartos]

Sort of.

We don’t precisely know the most fundamental workings of a living cell.

Our understanding of the fundamental physics of the universe has some hold.

But for LLMs and statistical models in general, we do know precisely what the fundamental pieces do. We know what processor instructions are being executed.

We could, given enough research, have absolutely perfect understanding of what is happening in a given model and why.

Idk if we’ll be able to do that in the physical sciences.

[wrs]

Having spent some time working with both molecular biologists and LLM folks, I think it's pretty good analogy.

We know enough quantum mechanics to simulate the fundamental workings of a cell pretty well, but that's not a route to understanding. To explain anything, we need to move up an abstraction hierarchy to peptides, enzymes, receptors, etc. But note that we invented those categories in the first place -- nature doesn't divide up functionality into neat hierarchies like human designers do. So all these abstractions are leaky and incomplete. Molecular biologists are constantly discovering mechanisms that require breaking the current abstractions to explain.

Similarly, we understand floating point multiplication perfectly, but when we let 100 billion parameters set themselves through an opaque training process, we don't have good abstractions to use to understand what's going on in that set of weights. We don't have even the rough equivalent of the peptides or enzymes level yet. So this paper is progress toward that goal.