While we should be careful of a bias, it is also a good practice in the scientific method to review definitions that may have been not precise enough.
For example, initially, a "planet" was just a big body in space. Then when people started to see more and more nuances, the definition just refined, and some objects stopped being called "planet".
I would not be surprised if there is a bias that pushes some people to redefine "intelligence" away from machine, but I would not be surprised if there is a bias that pushes some people to ignore newly discovered nuance and put into the same "intelligence" bag things that are in fact very different. I personally can see how LLM are not really "intelligent", and I don't think it is a good idea to say: well, yesterday we said the minimum criteria is X, now that we noticed that X can be reached without really doing the real thing, let's just ignore that and pretend it is the same thing.
(: the biggest clue for me is to use an early model, and see that it sometimes looks very intelligent, and then sometimes you can see that it gets it wrong in a way that shows that it never "understood" it at all. Newer models are better, but because it is an iteration on the same bases, the increase of performances cannot really due to replacing the things that "looked smart by aren't" by "real smart", but more replacing the things that "don't look smart" by "look smart by aren't")
Yeah I think if we are looking at it through that lens, the problem is in the term _intelligence_ in itself. Psychology and biology could not even pinpoint what exactly makes for _intelligence_. There isn't really a precise definition yet so it's just natural that definitions tend to shift.
I don't think we even need to go into tech and AI for an example. The intelligence or lack thereof of pets surprise us. Sometimes a cat is surprisingly smart when it is able to open a door to get food it wasn't supposed to. But then same cat gets bamboozled by walls and simple optical illusions. We generally expect that if something/a human is smart enough to do the former, then it shouldn't be dumb enough to fall for the latter.
Coming back to AI, this dissonance is how AI-generated images are detected for example. If a human can render something so well, you wouldn't expect them to make small but nonetheless elementary line art mistakes.
It's the same with human intelligence though. A human can be brilliant on some things and then we're puzzled why they are so idiotic in other areas.
Every time this comes up, people pick on any kind of flaws or inconsistencies of AI models, while at the same time giving a huge pass to the extreme variation in intelligence and stupidness displayed in human behaviour.
Creativity is the same. Human artists are "inspired" by earlier arts, perhaps following and slightly changing "trends" they participate in -- which is somehow seen as totally different from what AIs are doing.
No, this is not the same observation. In "basic LLM", the answer is not "confused" or "fail to understand", the answer is "inconsistent with the understanding mechanism". It is not that they "fail to understand while trying to understand", it is that there is not understanding mechanism at all.
Humans can have different level of intelligence, depending on the individuals, the subjects, even circumstantial situations (someone being tired, someone being distracted, or just bad luck). But they never make the same kind of mistakes I've observed with "basic LLM", where they do "non sequitur" that does not make sense at all but has all the characteristic of imitating something said by someone who understood.
I still even see it sometimes with Claude. It says logical stuff, and then suddenly something that does not make sense and it snaps me back to reality: none of this, including the correct things, are the result of understanding the underlying concepts, it is just that the correct things are more probable to generate, and that suddenly, a nonsensical happen to also be probable for a given configuration.
Humans don't make the exact same errors of LLMs of course. Humans are very different beings.
So you recognize that Claude is not a human.
Humans make mistakes as well "inconsistent with the understanding mechanism", but they have a very different form, and you are so used to the particular failure mode of humans, that you don't think about it.
But aliens visiting earth likely would find some aspects of human mind very peculiar!
Examples:
Humans learning algebra (or really anything like playing music, paddling a canoe, etc.) have to go through lots and lots of trivial basic mistakes, and only learn to avoid them through repetition and pattern matching on earlier experience, rather than relying on "reasoning".
A "pure reasonable being" would simply be learned the rules for algebra then go ahead and make perfect deductions applying the rules -- but humans are very clearly not such beings. Humans can know the rules for algebra perfectly well, then still go ahead and make mistakes until enough training has been done until we say you have "learned" it (be able to pattern match on previous experience).
Imagine humans being employed by aliens to do algebra, then aliens seeing humans basically do "2 + 2 = 5" (just on a higher complexity level). Like very human in first year in university WILL do with their formulas. What would you conclude about humans and their relation to "real understanding"?
Or another example: Humans engage a lot in post-rationalization, having first made up ones mind, then finding the reasons for the choice afterwards. (Most striking example of post-rationalization is the experiments on patients with severed brain half connections where one brain half invents a reason it can believe in for a choice made by the other brain half; https://en.wikipedia.org/wiki/Split-brain -- but if you look at pretty much any political issue for instance it is clear that people are driven at least as much by being herd animals as by doing any reasoning -- the majority of humans decide what people they belong with first, then figure out why afterwards).
I find this kind of reasoning a bit pointless and unfalsifiable.
Someone says "LLMs fail at this", and you say "but humans also sometimes fail", then they says "but we are not talking about the same thing", and you answer "this difference does not matter because aliens may be totally different".
My point is that what we observe with LLMs does not require any understanding. And in some cases, it is clear the answer of a LLM was built without understanding. And in other cases, it looks like it could have been built with understanding because there is no visible errors, but because we know the LLM can build things without understanding, this can equally simply be something that is built without understanding and happen to have no error, and therefore just looks like it has been built with understanding.
I think you take the problem the wrong way: you start from the hypothesis that there is understanding, and then you are finding reasons to maintain this conclusion (the most prominent ones being "humans also can do mistake" or "... fake understanding" or "... hallucinate". Well, humans can do a lot of things that don't require intelligence, does it mean that things that do these things that do not require intelligence are in fact as intelligent as humans?). This is a confirmation bias.
I don't have problem if it turns out LLMs have understanding. But the reality right now is that a simple explanation is that it does not have it. But it feels like some people just argue "but it is still possible, bending this argument there and there". I bet at some point, they will say "ok, I see your point, but maybe LLMs are intelligent and have this behavior on purpose because they want to remain hidden because they are smart enough to understand that if humans would know, they would freak out". It feels more and more like a belief system rather than a scientific approach.
Just two elements to go further:
- in the majority of cases, "things that have been faked to look like there are the result of understanding" will be correct. Because if you are trained to pretend you understand, you are trained to imitate someone who has understood, and you are therefore trained to imitate their reasoning, which turns out to be correct. (if you want to test the understanding, it is complicated, because the "understanding" is a data leakage during training)
- if LLMs extract understanding for the data from their training, it is strange that their current understanding (just after the training) is so close to the current understanding of the humans. Surely humans have missed stuffs here and there. The math theorem number 3424 not solved yet is probably as "simple" than the math theorem number 6423 that happened to be solved by humans, it is just by chance and circumstances that some humans have worked on 6423 and found a way to crack it while they did not spend as much time and effort on 3424. And yet, LLMs just happen to never notice any theorem on their own at the end of the training phase. Asking a LLM "Explain to me a math theorem that humans did not notice, with demonstration. This theorem should be something you understood when you were trained over maths" just does not work.
(and, please, I know that a mathematician may know theorem 3424 and yet not have noticed 6423 either, or that LLMs can scan a math problem with a large series of math tools and find a break-through. But my point is that LLMs are studying math soooo intensively during the training that they know all the human theorems, which is way more knowledge than any single mathematicians. And yet, it turns out that all this math understanding just ends up being exactly limited to what humans already know. What are the odds? Or more probably: they just don't really understand math, and when asked about a known theorem, they generate a correct explanation based on training data without properly understanding it)
I didn't really try to argue that LLMs "have understanding".
I am more, in a sense, arguing that "understanding" isn't clearly defined and sceptical about your confidence that this is an obvious quality of humans.
I'm not getting what this "understanding" thing is in humans that you are talking about. But I feel if anyone you are the one making the unfalsifiable statements here. You are the one talking about an inner quality within the reasoning that only humans possess and not LLMs.
If I re-read your post and replace the word "understanding" with "consciousness" then it makes a lot more sense to me. Yes, humans can be conscious that they understand something, while LLMs are very very probably not conscious of anything at all. If that is what you mean, I can easily agree with that. I would never argue LLMs are conscious.
But at least my original post had nothing to do with consciousness.
If I'm coding, I'd definitely pick the "understanding" of Fable over a junior engineer's "understanding" any day, for purely pragmatic reasons. When I say that, I simply mean that the rate of mistakes in junior humans is way higher than in best trained LLMs for most coding tasks.
I'm guessing this is not using the word "understanding" in a way you are happy with, and probably because you define the word "understanding" as being related to consciousness?
My problem with AI is the sheer variance of its stupid-smart spectrum. While it's true that human intelligence is not deterministic or predictable, the inconsistency exists in a much narrower band of variance which makes failure modes foreseeable. Thus I would much prefer a system with humans in the loop with processes in place for idiot-proofing.
This is true for "lateral" (I lack a better term) fields of intelligence as well. You don't ask a philosophy professor advice for the rashes on your skin; you see a doctor for that. And yet both the professor and the doctor could be expected to accurately identify from a picture that you do have rashes on your skin. An AI (and I mean in the general sense, not only transformer LLMs) could give you a pretty accurate rundown of Plato and still think the same picture is a beautiful sunrise.
(I don't even kid. Just this morning, an AI labeled a GIF from _Friends_ as a 1950s magazine ad for white bread. Just what in the failure mode is that?)
You can't idiot-proof AI without knowing what's in the training data set and even then you run into question of scale.
For example, initially, a "planet" was just a big body in space. Then when people started to see more and more nuances, the definition just refined, and some objects stopped being called "planet".
I would not be surprised if there is a bias that pushes some people to redefine "intelligence" away from machine, but I would not be surprised if there is a bias that pushes some people to ignore newly discovered nuance and put into the same "intelligence" bag things that are in fact very different. I personally can see how LLM are not really "intelligent", and I don't think it is a good idea to say: well, yesterday we said the minimum criteria is X, now that we noticed that X can be reached without really doing the real thing, let's just ignore that and pretend it is the same thing.
(: the biggest clue for me is to use an early model, and see that it sometimes looks very intelligent, and then sometimes you can see that it gets it wrong in a way that shows that it never "understood" it at all. Newer models are better, but because it is an iteration on the same bases, the increase of performances cannot really due to replacing the things that "looked smart by aren't" by "real smart", but more replacing the things that "don't look smart" by "look smart by aren't")