So we need to do more Good Old Fashioned AI, and get off my lawn.
He has a good point that model-less LLMs have serious trouble with problems that require a model.
But predicate calculus hasn't worked out well as that model.
Many years ago, I took John McCarthy's "Epistemological problems in artificial intelligence" class. He laid out the missionary and cannibals problem informally, then wrote it up in a predicate-calculus notation, and then applied his "circumscription logic"[1]. As he wrote it up in his formulation, I thought "and then a miracle occurs".[2] As with most story problems, it's getting the problem into the correct formalism that's hard. Turning the crank on the formalism is usually straightforward.
Much of the AI community spent the 1980s beating on this problem. A large number of very smart people tried to solve it. Many things were tried that were less rigid than predicate calculus - probabilistic logic, Markov chains, fuzzy logic, etc. All mostly failed. The AI Winter followed.
The classic critique in this area is "Artificial Intelligence meets Natural Stupidity", by Drew McDermott.[3] That's from 1976, and still relevant to this argument.
LLMs, though, might be able to use such models. Something to try: put a story problem into an LLM, and ask it what formal methods might help solve this problem. Then ask it to convert the problem into each of those formal methods. Then use something like Mathematica on each formal method. LLMs can't do logic problems, but they can sort of write code and translate between languages. So maybe they can do the miracle part. Anybody working on this?
I'm unsure if this is a meta joke or a great bit of irony.
> Anybody working on this?
If I understand your question accurately, yes. A more common example is people will ask GPT to answer via python code and then convert the python code into something else. But there are other people doing things more direct and through other methods. There are also people doing things like generating many answers, then performing search over those solutions (with or without GPT).
But regardless of, I think you should take care in calling out the "and then a miracle occurs"[0]. While the critique is well deserved, I think the context is dubious. It implies the same magic step is not necessary for LLMs. There's still a gap from where we are and getting to actual intelligence. LLMs are certainly impressive and have done a lot (something I think Gary ignores) but how to get to intelligence is still unknown and thus a missing middle step that "requires a miracle".
I don't think there is an issue in people pursuing neurosymbolics. In fact I would encourage it. Just as I'd encourage pursuing LLMs, category theory approaches, and others. The thing I would discourage is putting all our eggs in one basket when we recognize there is a missing step that we don't yet know how to solve. Allocate more resources to what's made the most improvements so far, but also not at the cost of recognizing limitations/criticisms. All technologies have limits and can be improved. It's the naive that reject critiques and the naive that are quick to dismiss. That's not science, that's politics.
> It implies the same magic step is not necessary for LLMs.
LLMs can at least take in an informal problem statement and do something semi-useful with it. This is progress.
Some other internal representations are needed, especially for dealing with the real world. Pictures? Animations? Animation storyboards? Graphs? Programs? All of the above? Ten years ago it would have been a fantasy to consider pictures and animations as intermediate representations in an AI system. Not any more.
Automated storyboard generation is already a thing.
I want to stress that not only do I agree with this, I explicitly stated so. I even explicitly said it would be naive to dismiss this progress AND explicitly criticized Gary for doing so. I want to make it abundantly clear that I am not claiming LLMs are not progress. I feel I have to do this because the context here and because it is common to conflate critiques of LLMs with dismissal of LLMs.
> Ten years ago it would have been a fantasy to consider pictures and animations as intermediate representations in an AI system.
I'm hesitant to agree. I'll agree if we are also saying that 10 years ago it would be considered fantasy to build a lossy compression of human written knowledge, build a natural language interface into it, and have this all under 200GB. In that I think someone could imagine a system but think it is far away and maybe even not believe the last condition. But this is a reasonably accurate description of LLMs (what's up for debate is the reasoning capacity, not the compression aspect).
And my point is not about technical capabilities. Like every other ML researcher, the release of GPT made me believe AGI was much closer than I had previously thought. But similar to many ML researchers, I later again reevaluated returning back to a similar position as I transitioned from seeing examples to have intimate experience with usage, deeply diving into the data they are trained on, into the training processes, and probing these machines.
For some people understanding the mechanics behind a "magic trick" makes the magic trick unimpressive. But I've always been fascinated and the mechanics often makes the tricks far more impressive! What GPT made everyone reconsider is how much we could do with data alone. How powerful and impressive our existing statistical frameworks are when scaled. But there is no evidence here that these systems actually understand what they are processing. There is no evidence that these systems are logically reasoning and there is a fair amount of evidence that they are not[0]. The details here matter and are the critical part of answering these questions. Because, as you mentioned, we've made a lot of progress. And the thing is that when we progress, the amount of complexity needed to further advance also increases. A low order approximation takes you a long way but we know complexity increases quickly to increase accuracy slowly.
I guess I would be more willing to believe a path to AGI argument with the systems if they were more robust (not to say I am not still impressed). I think even if the systems could perform the image generation tasks I described here[1] (see Imgur link), I do not believe this is enough to demonstrate intelligence or reasoning, alone. The types of errors made are not illustrative of a system that understands but it also is important to remember that proof is not symmetric (note: image generation is my specific research area). A billion positive examples do not constitute a proof while a single counter example constitutes a counter proof. My concern is that these discussions are often in the form of demonstration as proof. Demonstrations aren't proof and no amount of them will constitute proof. But it's also important to note that a counter proof is not always an _absolute_ rejection but often is more often bounding. What I'm saying is that the counter examples don't dismiss the utilities of LLMs but they do place strong bounds on where the utility lives. The distinction does matter, and a disregard of this distinction is specifically what I am criticizing Gary for.
I want to also mention that the word "reasoning" is not always constant and that this unfortunately makes the discussion more convoluted. But I think we need to understand what is in the training data to accurately understand how to accurately test these abilities. Similarly the terms "out of distribution" and "zero/low shot" are changing and often not in great ways. E.g. it is common to train on LAION and "zero-shot" on ImageNet.
Perhaps a good way would be to define a logics language for AIs or use an existing one like Prolog and let the LLM generate code then run it. It's a variant of giving LLMs access to some system and let them iterate till they find the solution.
The idea: When programming sometimes the solution is not exactly right but with feedback a better solution can be found. I once made that explicit to the LLM and I played the evaluator for the LLM and it seemed to work better. I am not surprised. Even humans usually don't just program on paper. In the early days of the computer science they had to do that. I experienced that myself: my first BASIC program was on paper and then when I had access, I typed it in. The experience is bad. I had to guess without feedback.
I can imagine that it doesn't solve all problems because not all is solvable by trying out things or the formulation of the problem doesn't describe the problem correctly. But it is better than not trying out things at all.
Please see Animats comment: https://news.ycombinator.com/item?id=41095417 and the answers there. They are saying more or less the same thing but more detailed and knowledgeable than me.
There are a number of ways the future could go but comparing language models like chatGPT to human thinking the more obvious way forward might be visual reasoning and spatial modeling.
A lot of human thinking is vision related as in 'I see what you mean' 'picture this' and so on. Also comparing current AI to humans it's getting quite good at written exams but terrible at physical stuff like getting some milk from the shop and making a cup of coffee. Also the emergence of something like physics in the likes of SORA suggests it's possible.
On the other hand symbolic logic along the lines of algebra is quite a specialist area that humans have to be taught in maths classes and many get by without learning. I presume AI will get good at it but it doesn't seem the most obvious way forward unless you want it to do maths.
(by the way I came across a paper on trying to go this way https://spatial-vlm.github.io/ - adding 3d data to a multimodal large language model)
I guess human reasoning has three major areas - language, visual/spatial and also thinking of other thinking entities like other people, the cat wants to go out etc. Likely consciousness relates to the last category - does he feel like I feel and the like.
I tried the goat puzzle on chatgpt it it got it right with reasoning as below. This suggests to me the systems are improving and don't seem at an obvious plateau yet.
>To solve this puzzle, it's important to make certain assumptions and understand the conditions. Since you mentioned only a man and a goat and didn't specify any constraints, I will assume the following:
1 The boat can hold both the man and the goat at the same time.
2 There are no restrictions on how the man and goat can travel in the boat.
3 The goal is simply to get both the man and the goat across the river.
Given these assumptions, the solution is straightforward:
The man and the goat get into the boat.
They both cross the river together.
Therefore, both the man and the goat can successfully cross the river by traveling together in the boat.
Note that the hot patching may not be explicitly driven. As in there isn't someone looking at the failure of specifically goat problems and specifically seeking out to improve it. But rather that data is continually ingested and the more viral a limitation becomes, the more likely it is to be captured in the new training. Though that isn't to say that someone won't throw in a few river crossing puzzles to the RLHF part of the training. Both these will make the model better at these puzzles but not make them better at the abstract capabilities people are using these examples to show that LLMs are incapable of.
The best thing to do, and I can assure you still works, is to use variations. In fact, this is something both Colin (Fraser) and I have been doing over the last few years. The point of them isn't to prove that the models are incapable of solving the puzzles, it is to show that they are brittle. It is to show how subtleties can cause failure in environments where we know what the correct answer should be so that we are cautious in environments where the solutions are unknown (at least to us).
Here is an example of such a variation[0]:
> A farmer must cross a river with a goose, a snake, and a duck. Only one animal may fit in the boat with the farmer. If the farmer leaves the snake alone it'll slither away as the goose or duck attack it. Both the goose can fly and swim and but the duck can only swim because its wings are clipped. They will follow the farmer wherever he goes. What is the minimum number of trips the farmer must take to get all animals across the river?
I'll also mention that my first go had "Both the goose and duck can both fly and swim" with which it was able to get the right answer. Note that my modification does not realistically change the outcome, but it causes GPT to answer differently (in my case it thinks 5 trips while believing the duck can fly). It isn't that I iterated to find something that "tricked" GPT (even if my second try), it is that the ability to do this demonstrates that the machine does not actually understand what is being asked[1].
[1] Yes, humans may fall for similar tricks. But usually for a different reason, specifically they are thinking you are trying to trick them and are looking for the trick. GPT isn't expecting a trick and is treating the question at face value. But if you keep doing this with humans (implicitly done in GPT training) they'll learn there is no trick and then get 100% accuracy. Their error will then be more likely in that they'll turn off their brain and if you switch back to tricking they'll error until they reorient because they were allocating processing power elsewhere. GPT is giving you 100% all the time, humans do not.
Not sure this really holds: there is no symbolic LEAN type system in any animal brain as far as I can tell. I do think the point about the AI needs to sanity check is right. I imagine all this is will fall out from the visual, audio and other senses feedback loops though. I dont see the fundamentals needing explicit algebraic reasoning - that will surely come later.
The brain doesn't have a CPU-like architecture doing symbolic reasoning. In fact, we now offload this kind of reasoning to computers because they do. If you want to create human-level AI, this is not the way. You'll be able to create really powerful and useful niche systems like Deepmind has, but not a truly general reasoning machine.
> The brain doesn't have a CPU-like architecture doing symbolic reasoning.
While I agree with the first part I'm not convinced of the second. Can you elaborate? I'm fairly certain people write down symbols on paper (or chalkboards or computer screens), manipulate those symbols, and then uncover new laws of physics[0].
> we now offload this kind of reasoning to computers because they do.
Do we? I'd also like this elaborated on. I do not disagree that we offload computation, but I'm not convinced we offload reasoning. There are automated reasoning systems but they are still human directed and I should mention are symbolic in nature.
I also want to make sure we're using the word "reasoning" to refer to the same thing. I am using the definition from here[1] which is congruent with this[2]
All reasoning requires self-awareness, how can you cause electric circuits to become self-aware when you can't replicate the "awakeness" that is intrinsic to all sentient beings?
In other words, how can AI tap into the hyper-dimensional knowledge plane that we access with our consciousness first and then vocalize into sounds and letters?
You can RAG what is written only but not what was "thought" to write those letters and so you cannot have a purely rationalizing and reasoning electrical circuit that function like the biological brains.
It's like feeding magic mushrooms to a robot and expecting it to become conscious. There is no bio-chem pathways to activate the pineal gland nor is it possible to mimic it even.
I glanced at that and it doesn't seem very good. It mentions AGI fifteen times but doesn't say how it defines it. It hints that LLMs don't count because they don't do original reasoning but doesn't give much argument why different algorithms can't do that.
I read it too and it seems like the problem is precisely that nobody has a precise definition of what AGI will be. A system that is always right ? If so impossible. A system that is just like humans ? Humans are sometimes right and sometimes wrong so how do you manage it when it is a machine with zero consciousness? At least the article correctly explain why we can't build AGI with LLMs only.
I think people have personal definitions but they differ so it can be hard to know what you are talking about. For me I have it as able to do all the intellectual stuff that humans do including making scientific discoveries, designing new computers and so forth.
AGI doesn't really specify a level, like Deepmind's general video game player was artificial and general but most people think human level or better with AGI.
This keeps emerging again an again, and the answers are pretty generic.
1. Large language models as a concept are not going anywhere anytime soon for any reason. Simply because there's no other source with a huge slice of human psyche encoded into it than the language itself and the corpus of texts in it. Humanity collectively did a massive amount of gradient descent on the language over generations, and it will stay as the primary source. That doesn't mean that other sources don't exist, of course.
2. Dataset quality matters at least as much as the architecture. There's plenty of low-hanging fruit available in preprocessing the data and "textbooks for models". You learn to count in a decimal system from both memorizing the number sequence and the explanation of the algorithm, not just by looking at millions of examples! There's plenty of a bit higher-hanging fruit available in hardware improvements and optimizations.
3. Calling a transformer a token predictor, stochastic parrot, autocomplete on steroids, etc. is of course right but kind of misses the point, like calling human brain a nerve impulse predictor (and the brain also has no "inherent way of verifying whether their predictions are correct", using the definition from the article). Reasoning about this in ill-defined terms like "understanding" or "knowledge" or "intelligence" is not useful at all. There are many differences between humans and LLMs, but the most high-level one is that humans are autonomous agents that exist in continuous time, and transformer's lifetime is the time required to compute a single token. Repeat the process for multiple tokens and you have something more complex. Add an external loopback, and you have a chatbot with memory, partly capable of doing things unexpected of a "word predictor". Make the loopback more complex, and you suddenly have an... autonomous system that exists in continuous time. Sure, it's extremely crude and primitive, and that loopback probably also needs to be replaced by something way more advanced in the future, and, and, and, and...
4. Reasoning and symbolic computation comparable to human abilities (which are also pretty spotty and error-prone) might or might not emerge as a result of scale and simple loopback mechanisms in models. You might or might not need an external symbolic engine as the author says, or maybe you can reduce it to another model of a different type, or maybe it's all wrong. Current models are still orders of magnitude smaller and simpler than the human nervous system, and plenty of things in LLMs already changed by simply increasing the scale.
5. Other than all of the above, sure - transformers or another flavor-of-the-year architecture might give way to more advanced ones. But the basic principles will remain, and language models are not going anywhere.
> huge slice of human psyche encoded into it than the language itself and the corpus of texts in it
This type of wording is problematic because it conflates what is written as representative of our psyche when it does not.
Psyche implies thinking and thoughts that have occurred when the brain accessed those concepts from outside our 3D world, processed it internally, vocalized it into sounds and then finally letters.
LLMs are just doing pattern text search on top of what is written, it is doing no sort of reasoning or accessing of the hyper dimensional plane like our brain does when it thinks or reasons with concepts.
Our brains are not some exotic token or neurosymbolic search engines!
Differences in particular representations or the generation process are not very interesting, what matters is the stuff encoded in it. And as I said, calling a model a token predictor is right, but kind of misses the forest for the trees, it's an argument on a lower abstraction level that is not very useful.
The biological capabilities of a single human are also not very impressive, by the way. 90% (made up number) of what you consider your intelligence is actually the result of the biological evolution and social processes accumulating and abstracting the knowledge over endless generations. Hypothetical you raised without any contact with other humans, society, culture, education will be substantially different. So the processes are not just in your brain.
Whether you or me are doing "reasoning" is the matter of definition, and it's a really vague term. If you try to define it with more precision, you might come up with an idea that all we do is post-rationalizing the result of our blind prediction.
> This type of wording is problematic because it conflates what is written as representative of our psyche when it does not.
It definitely is representative, in some way. Human civilization did a huge amount of combined computation to encode the human behavior (personal, social, all kinds) into abstractions/semantics hidden in the language and text. Surely it can be recovered with some precision by statistical analysis and some computation. Which is what a large language model does.
Of course this "reverse engineering" approach has limitations. The model might not be able to generalize well enough to pick up higher level semantics. It might be architecture-limited. Some data might just not be in the dataset. The model will never be able to 100% copy humans without having an extremely precise biological reference, as well as you'll never be able to copy a dolphin, alien, or a model. But having an artificial human is not the point of this, and the achievable precision might be just good enough.
you are fixated on the output and what we can do within that limited set of data while ignoring the thought processes that was involved behind outputting that data as well as interpreting it.
without that "thinking" portion and simply mimicking to the point it resembles it while no such activity is happening (as I defined as accessing the conscious hyper dimensional cloud we humans can do easily).
intelligence in the english vocabulary is limited to retrieval which seems to be why there is so much push towards LLMs but this like trying to dance to a painting, you can interpret it as music and mimic dance moves but its different when a human hears the music and moves naturally.
That sounds... suspiciously like attributing inherent magic to humans. Who cares in practice that mechanisms under the hood are different if the output is the same? Why exactly are you sure humans do all these vaguely defined things inherently, and that it's not an emergent property? What even makes you think you do that? Individual capabilities are vastly overrated.
And the goal is not copying humans to begin with, just interfacing with them and be aligned just enough.
To me neurosymbolic seems to be a mostly tribal distinction. Also Gary Marcus is just trying to make Google's achievement all about himself again because he's a narcissist. But what else is new
You are a fool, if you think it is a “a mostly tribal distinction”. You would have said same about alchemy vs chemistry, or genes as proteins vs genes as DNA. In science, the details matter.
I mean he's free to hold whatever political opinions. But yeah... he is a contrarian hater usually childish and I hate the sensationalist manner he uses to present things, everything is insanely important and we must act *NOW* and all that. Ever since he was apart that hearing at congress I now view every Gary Marcus tweet through the lens of him desperately wanting to become part of some transnational ai oversight organisation. I just hope he doesn't get any power
It seems like you mainly switched to anti-neural-network punditry. Have you thought about trying to go back to research or is that not viable for you anymore?
I know this is a contentious subject and tbh I'm not a big fan of Gary myself, but I'd like to make a simple explanation as to why AGI requires symbolic manipulation. Note that I'm careful here and not saying it needs to completely neurosymbolic nor am I saying the advancements we've made won't contribute to AGI (they will). Unlike Gary I am not willing to ignore the advancements that LLMs have made and the utility that they provide. Their limitations do not invalidate the improvements. But this is the identical error those make that dismiss symbolics as an avenue forward.
At the end of the day, the current framework of ML operates by ingesting data, modeling that data, then iterating on that model to better fit data[0]. This is something humans and every living creature does. But many animals, humans included, do so much more. The problem comes when we get to understanding abstraction, and an issue is that we operate at such extreme levels of abstraction that it is easy to miss. After all, we were designed to be better at recognizing differences because it allows energy savings for business as usual settings. The problem is that pattern recognition leads to no such viable path for our levels of abstraction. It may not be obvious, but we are symbolic manipulators. Our language is composed of symbolic manipulation, our code, and our math. The last of which may be the most clear example of the distinction. But this might be a bit uncommon to see unless you're from a strong science background.
We all know that there are great differences in numeric/empirical solutions from analytic solutions. The latter of which is considered both harder and much more rigorous. The latter is naturally causal and interpretable. The reason we do numerics and empirics is because limitations. But analytics is why a physicist can sit in a room with a pen and paper and (eventually) discover fundamental laws of nature. Many of these achievements are not solvable by observation alone[1]. But these equations are symbolic. The symbols are the abstraction. A major advantage of the symbolism here is that once we are able to formulate solutions and the rules of the symbolic system, we can manipulate as we please. This has so much more flexibility than a numeric solution. This is the underlying reason the theorist exists! It allows for us to quickly and accurately ask new questions and find errors or limitations. This kind of manipulation allows us to ask why gravity is an inverse square law and to understand why it is exactly 2[2] and not close to 2. It allows us to set concepts aside that we might call a constant (when the resultant is unitless), solve currently tractable factors, and even then later determine what this constant is (often the job of an experimentalist). We may even later ask ourselves how a constant may be decomposed into other factors. The symbolic nature allows us to pattern match in ways we wouldn't be able to with numerics. There is just a high level of abstraction that we are unable to do with the numerics. Abstraction that we rely on to create and understand the world as it is.
So the great question in AI/ML is not if these systems need to do symbolic manipulation. It is if a machine can learn to do symbolic manipulation through numerics. This is still unknown and there are arguments on both sides (right now the case is stronger against this happening through data processing). The only naive thing would be to not pursue both paths (well there are many paths). We're venturing into the unknown. We've gotten a long way through the methods we've been using and this is reason to continue down that path. But at the same time, this is not reason to pursue others. We've never seen that happen in the past. All technologies undergo radical shifts that are no apparent to those on the outside. Imagine if we didn't pursue LiPo because lead-acid was working. If we didn't pursue transistors because vacuum tubes were working. LEDs because incandescent. And all the new technologies began as worse (often much worse) than those they later replace[3].
Lastly, I want to speak to investors directly. If your goal is to invest in a new company that will make AGI, you are likely to lose if that company is pursuing via LLMs[4]. There are already major players in this space that are far ahead and have more momentum and funding. There are things that they are missing that others might see, but they have the capacity to find those limits and fix them[5]. Instead, you have a better chance on what appears riskier: less developed avenues that also have explainable avenues towards the goal. It is high risk, but it always was. Here's the thing, I said the question of numerics leading to symbolics is still open, but another way of looking at this is that we know symbolics is necessary (or at least we know symbolics is sufficient for intelligence).
[0] Note the dependence on the previous estimate/model.
[1] I want to note that there is a feedback mechanism which is what I reference in my first paragraph. The theoretical physicist stands on the back of experimental physicists just as the experimental physicist stands on the back of the theorist. An untest{ed,able} hypothesis is no theory and a fitting data is not a physical model without theory. See Fermi/Dyson's conversation about fitting an elephant.
[2] This is a calculation you will do in an upper division classical mechanics course (physics).
[3] Often also with people questioning why we should pursue these other paths, not recognizing -- or unwilling to -- the limitations of the current technology. And no technology is without limits. That alone should be reason to pursue other avenues.
[4] If you're investing in products, then pursue LLMs. They are much more mature and you have the infrastructure of research behind you. You can also likely adapt to a changing underlying technology.
[5] Unless your real goal is acquisition by said players
He has a good point that model-less LLMs have serious trouble with problems that require a model. But predicate calculus hasn't worked out well as that model.
Many years ago, I took John McCarthy's "Epistemological problems in artificial intelligence" class. He laid out the missionary and cannibals problem informally, then wrote it up in a predicate-calculus notation, and then applied his "circumscription logic"[1]. As he wrote it up in his formulation, I thought "and then a miracle occurs".[2] As with most story problems, it's getting the problem into the correct formalism that's hard. Turning the crank on the formalism is usually straightforward.
Much of the AI community spent the 1980s beating on this problem. A large number of very smart people tried to solve it. Many things were tried that were less rigid than predicate calculus - probabilistic logic, Markov chains, fuzzy logic, etc. All mostly failed. The AI Winter followed.
The classic critique in this area is "Artificial Intelligence meets Natural Stupidity", by Drew McDermott.[3] That's from 1976, and still relevant to this argument.
LLMs, though, might be able to use such models. Something to try: put a story problem into an LLM, and ask it what formal methods might help solve this problem. Then ask it to convert the problem into each of those formal methods. Then use something like Mathematica on each formal method. LLMs can't do logic problems, but they can sort of write code and translate between languages. So maybe they can do the miracle part. Anybody working on this?
[1] https://en.wikipedia.org/wiki/Circumscription_(logic)
[2] https://trevor-hopkins.com/fiction/miracle2.jpg
[3] file:///home/john/Downloads/Artificial_Intelligence_meets_natural_stupidity-2.pdf