[romanhn]

"People say, It’s just glorified autocomplete ... Now, let’s analyze that. Suppose you want to be really good at predicting the next word. If you want to be really good, you have to understand what’s being said. That’s the only way. So by training something to be really good at predicting the next word, you’re actually forcing it to understand. Yes, it’s ‘autocomplete’ — but you didn’t think through what it means to have a really good autocomplete." - Geoff Hinton

[kypro]

The definition of "understanding" is something people disagree on which doesn't help when having this conversation. Although I agree with Hinton his quote here could apply to a lot of things most people wouldn't ascribing understanding to. For example, a calculator is really good at maths, so does it therefore have to have a really good understanding of maths?

I take the opinion that a simple autocomplete, like a simple calculator does understand something about the world, but their understanding of the world is extremely narrow. A simple autocomplete "understands" at the very least simple relationships between letters and words. That doesn't mean a simple autocomplete "understands" anything about the meaning beneath those words, but it does understand something about how letters and words are often used. Similarly a calculator "understands" simple mathematical operations, but nothing beyond this making it's understanding of maths is very good within it's domain, but extremely narrow.

When you start adding additional breadth to the understanding then I think that's when you edge closer to human-level understanding and it's that breadth when combined with some amount of depth that most people associate with a "true" understanding. Adding context to a simple autocomplete, ie, "cat is related to dog" provides a depth of understanding about the relationships between words beyond the that of a simple autocomplete. If you keep adding more context (relationships between concepts) you approach something more like the understanding humans process.

I guess what I'm saying is that the primary problem here is that most people define "understanding" as something very human so if a LLM doesn't understand the world similarly to us then they reject that they understand anything. This debate first requires we define what it means to understand, and in my opinion any workable definition would start with us agreeing that a calculator genuinely understands mathematical operations.

[kromem]

A calculator does have an excellent understanding of math.

In that case, it's an understanding directly programmed in by developers who have an excellent understanding of math.

In the case of a LLM, there is no direct programming of any understanding, and the version best able to predict next tokens developed its own 'understandings.'

The problem is that when sufficiently complex, we really have no idea just what those understandings are, so it could be "this word often goes after these other words" or "given the context I should be happy and a happy person would say this."

Those are two very different levels of understanding, and while research over the past year has pretty well demonstrated that at least some world modeling in linear representations is occurring, those findings are in toy models and something as complex as GPT-4 is a giant black box where what % if understandings are surface statistics and what % are something more is pretty much a giant question mark.