| Ok, I've stopped reading half way because it is useless. Your reflection does not bring anything concrete, it is just you trying to play on semantic and loose the point. For example, you just moved the definition problem to "novel". Do you even realise that? You are claiming that there is an understanding because the model is able to do something in a novel situation where only deeper understanding of the situation will allow it to perform that well. The big problem is that you have no idea if this situation is "naturally easy to reach" or not. For example, a system that is fitted on the electrostatics Coulomb's law will build, internally, a set of equations to generate realistic predictions. And then you take this system and put it in a totally novel situation: classical gravitational problems. Well, this system will be able to generate realistic predictions there too, because Newton's law works with equations that have the same form. When you are discussing with a LLM in a "novel" subject, how do you know the LLM cannot directly use the fit and complex equations that it has created for the "non-novel" situations it has been trained on? For example, the LLM has been trained on "Pride and Prejudice and Zombies" and tons of other mash-ups. Even if asking a story of Keanu Reeves and the Supreme Court looks "novel" to you, it does not mean the generated text was not in fact super easy to generate based on the patterns that the LLM has seen in tons of examples. Honestly, this whole conversation just convinced me that too many people who claim "GenAI does understand" are way above their head on the subject. If you want to continue talking, just talk to a LLM. Plenty of people have done so and convinced themselves they were geniuses when in fact they were not at all. Yet another example that LLM has no understanding, as it is very very often failing to distinguish between correct ideas and "things that look correct but that someone with real understanding will not encourage". |
"Novel scope" just meant the scope beyond training data, discoverable by experimenting, that a post-trained model was able to generalize well to.
It didn't mean arbitrary or alien to training data.
Thesis: The greater a model's scope of generalization, the greater evidence for "understanding" instead of fitting. I can't think of a better way to compare levels of understanding, for models of comparable size, than by how far each of them can generalize beyond training data.
I didn't always follow you either. But I didn't think you were being flippant or unreasonable when I didn't.
No worries. I appreciated being pushed to think more clearly, and you made points that improved my thinking directly.
EDIT ——— I think trading walls of text was a challenge. It seemed sensible to try and respond to "everything", but I can see that one specific at a time would have worked better. And I need to find someone in my vicinity to bash ideas with. I have settled in a new area, and miss that. So thanks.