[maxwells-daemon]

The "language models don't really understand anything" corner is getting smaller and smaller. In the last few months we've seen pretty definitive evidence that transformers can recombine concepts ([1], [2]) and do simple logical inference using contextual information ([3], "make the score font color visible"). I see no reason that this technology couldn't smoothly scale into human-level intelligence, yet lots of people seem to think it'll require a step change or is impossible.

That being said, robust systematic generalization is still a hard problem. But "achieve symbol grounding through tons of multimodal data" is looking more and more like the answer.

[1] https://openai.com/blog/dall-e/ [2] https://distill.pub/2021/multimodal-neurons/ [3] https://openai.com/blog/openai-codex/

[6gvONxR4sf7o]

> The "language models don't really understand anything" corner is getting smaller and smaller.

In my mind, understanding a thing means you can justify an answer. Like a student showing their work and being able to defend it. An answer with a proof understands the answer with respect to the proof it provides. E.g. to understand an answer with regards to first order logic, it'll have to be able to defend a logical deduction of that answer.

These models still can't justify their answers very well, so I'd say they're accurate but only understand with respect to a fairly dumb proof system (e.g. they can select relevant passages or just appeal to overall accuracy statistics). They're still far from being able to justify answers in the various ways we do, which I'd say means that by definition that they still don't understand with regards to the "proof systems" that we understand things with regards to.

Maybe the next step will require increasingly interesting justification systems.

[krisoft]

> In my mind, understanding a thing means you can justify an answer.

Do you understand cats? If I show you a picture of either a cat or a dog do you think you can tell which one it is? I think most people could solve that challenge, and if pressed they could vax poetically about what makes them think it is a cat. Maybe they would mention the shape of an ear, or talk about feline grace or what have you. But is that really a “justification”? Let alone one they can “defend”? How would “defending” even work in this situation?

[hayksaakian]

You could probably teach an AI to post-hoc rationalize their decisions, the same way people do.

[taneq]

You absolutely could, and it could even end up just as accurate as human post-hoc rationalization. ;)

Self-analysis and self-interpretation is pretty clearly a key part of consciousness... I do wonder - how important to the process is the actual fidelity of the interpretation? Those people you meet who think they have deep insight into their own psyche while clearly having no clue... maybe they're p-zombies. ;)

[naresh_xai]

That’s basically the gist of explainable AI

[6gvONxR4sf7o]

The point I’m trying to make (poorly) is that i don’t think a one size fits all definition of “understanding” is useful. It’s more useful to define understanding with respect to how you can justify a thing you know.

So for the case of cats, I will understand cats at a different level from a cat biologist. I can point to features that seem catlike, and they can talk about all sorts of other scientific things that make a cat a cat.

With respect to that sciency kind of understanding, I don’t understand cats. With respect to a much looser ‘point at the features’ kind of understanding, I do understand cats.

[Cybiote]

Take entomologists, bird watchers or those who identify mushrooms. In each, there are instances that are fiendishly difficult to tell apart. If you ask an expert for advice, they'll tell what features to look for and where, sometimes not even on the item itself and some requiring specialists tools.

While explanations are far from sufficient to instantly close the gap to expertise, they provide a massive boost that you might not otherwise have found on your own. The justification comes from the fact that their explanations are a reliable source of increased performance in making fine-grained distinctions. It's further demonstrated by answers to questions like "If they are so difficult to tell apart, why make these distinctions?" or "How did they come to be so similar?".

[beering]

> In my mind, understanding a thing means you can justify an answer.

What if the language model can generate a step-by-step explanation in the form of text? [0]

There's no guarantee that the reasoning was used to come up with the answer in the first place, and no proof that the reasoning isn't just the product of "a really fancy markov chain generator", but would you accept it?

We're really walking into Searle's Chinese Room at this point.

[0] https://nitter.hu/kleptid/status/1284069270603866113#m

[naresh_xai]

Umm, no there are clear verification methods for Explainable AI techniques today. One way to check the justification would be if things which were important in the justification were removed in some sense, then would the output change signficantly. Sort of like a sensitivity test for justification.

[6gvONxR4sf7o]

Searle's Chinese Room is exactly why I started thinking of understanding this way. It convinced me that a one-size-fits-all notion of understanding isn't useful. But it also made me think that understanding "with respect to system X" is useful.

If you can challenge an answer and get justification expressed in the form of X, then it understands with respect to X. A step-by-step text explanation is one form of X.

> ... but would you accept it?

This is all to sidestep questions of whether you accept X as "real" understanding or not. :D

[piyh]

>There's no guarantee that the reasoning was used to come up with the answer in the first place, and no proof that the reasoning isn't just the product of....

You're holding machines to a higher standard than we hold people.

[maxwells-daemon]

Look at the "math test" video.

Given the question: "Jane has 9 balloons. 6 are green and the rest are blue. How many balloons are blue?" The model outputs: "jane_balloons = 9; green_balloons = 6; blue_balloons = jane_balloons - green_balloons; print(blue_balloons)"

That seems like a good justification of a (very simple) step-by-step reasoning process!

[krisoft]

I wonder what would it have outputted if we would remove the “ and the rest are blue” part from the question.

Would not surprise me if an innatentive human student would answer that with the same code. After all school “trains” people to expect such challenges to be solveable. A more attenive human might say “we can’t know” or provide an upper limit to the number of potential blue balloons.

[spyder]

Related article: Teaching GPT-3 to Identify Nonsense

https://arr.am/2020/07/25/gpt-3-uncertainty-prompts/

[riku_iki]

chances are high that something similar was in training set, and model approximated it.

[krisoft]

You are very likely right. The question is how far the approximation can generalise? One way to test that would be to quizz the model with slightly varied prompts. Any human who can “solve” this world problem should be reasonably expected to solve the same problem if we change the subject’s name. ( From Jane to Bob, or Sanj, or even to Xcfg.) Or the name of the object (From balloon to token, or even to embobler). Or the attributes used to segment them. (From red/blue to heavy/light for example)

Or we can try to rewrite the challenge sentences with different wording. As long as the new sentences convey the same problem you would expect that a system who can “understand” them would generate the same or similar solution.

Curiously this kind of thought experiment also shows a weakness of the Turing-test as originally formulated. A machine correctly solving these word puzzle variations could “prove” that it “understands” the sentences, but it would also reveal that it is not a human. Since i would expect a real human to protest against the inanity of the challenges quite fast. ;)

[FeepingCreature]

This goes for humans too. Ultimately, "something similar was in the training set" is semantically indistinguishable from "having a rich generalizable conceptual toolbox".

[wizzwizz4]

Except I could do that with a few regex substitutions, which would not be reasoning. The “intelligence” is in the templates provided by the training data. (Extracting that is impressive, but not that impressive.)

[sbierwagen]

>In my mind, understanding a thing means you can justify an answer.

Sure, but how does that work with superhuman AI? Consider some kind of math bot that proves theorems about formal systems which are just flat out too large to fit into human working memory. Even if it could explain its answers, there would just be too many moving parts to keep in your head at once.

We already see something this in quant funds. The stock trading robot finds a price signal, and trades on it. You can look at it, but it's nonsensical: if rainfall in the Amazon basin is above this amount, and cobalt price is below this amount, then buy municipal bonds in Topeka. The price signal is durable and casual. If you could hold the entire global economy in your head, you could see the chain of actions that produce the effect, but your brain isn't that big.

Or you just take it on faith. Why do bond prices in Topeka go up, but not in Wichita? "It just does." Okay, then what was the point of the explanation? A machine can't justify something you physically don't have enough neurons to comprehend.

[6gvONxR4sf7o]

It's not about us being able to interpret answer or justification, but the reasoner's ability to justify. If a superhuman AI can justify its answers in terms of first order logic, for example, it could be defined as understanding the answers with respect to FOL. Whether we as humans are able to check whether this specific bot in fact meets that definition is a separate empirical question.

If that quant algo you mentioned just says "it'll go up tomorrow" that's different than "it'll go up tomorrow" with an attached "it's positively correlated with Y, which is up today" which is different from a full causal DAG model of the world attached, which is again different from those same things expressible in english. But again, those are definitions, which are separate from our ability to check whether they're met.

Luckily, we're not in the realm of bots spitting out unfeasible to check proofs, except for a few niche areas like theorem proving (e.g. four color theorem). For language models like in the article, the best I'm aware of is finding relevant passages to an answer and classifying entailments.

> A machine can't justify something you physically don't have enough neurons to comprehend.

We can't always verify its justification, but it either can or can't justify an answer with respect to a given justification system.

[gnramires]

Also, you should note the memory and capabilities required to reach a conclusion might be much greater than to show it's true. Showing a needle may be easy, finding it in the haystack very hard. In this sense the hope for explainability is expanded. But still, I guess the real world is really messy "the full explanation" may be too large -- like when you explain a human intuition, the "full explanation" might have been your entire brain, your entire set of experiences up to that point; yet we can give partial explanations that should be satisfactory

A have a hypothesis that inevitably, reasoning needs to 'funnel' through explicit, logical representations (like we do with mathematics, language, etc.) to occur effectively. Or at least (quasi-)formalization is an important element of reasoning. This formal subset can be communicated.

[gnramires]

> Even if it could explain its answers, there would just be too many moving parts to keep in your head at once.

While this is possible in practice, consider the (universal) Turing machine principle: in principle, you can simulate any system given enough memory; we may not have it our brains, but we have pen and paper or simply digital text scratchpad (both of which we use extensively in our lives).

[cscurmudgeon]

We build another system we fully understand that can process the justification and see if it is correct/makes sense.

[hakuseki]

What about GPT-f? It's a language model that proved theorems in the metamath formal system.

[6gvONxR4sf7o]

I'd definitely say it understands those theorems with respect to the metamath formal system then. The next question is what it understands the proofs with respect to.

[visarga]

> Maybe the next step will require increasingly interesting justification systems.

You can just ask it to comment what it intends to do. It's surprising actually.

[joshjdr]

I found it on Stack Overflow!

[karmasimida]

> The "language models don't really understand anything"

This is still true. By all account, human doesn't need to read 159GB of Python code to write Python, or we simply can't.

But it doesn't necessarily indicate language models aren't useful.

[hackinthebochs]

Considering the sum total of data and computation that goes in to creating an intelligent human mind, including the forces of natural selection in creating our innate structure and dispositions, it's not obvious that any conclusions can be drawn from the fact that so much data and compute goes into training these models.

[nightski]

Has this transfer of knowledge from one domain to another really been demonstrated by these models/learning processes? I know transfer learning is a thing (I have a couple books on my shelf on it). But it seems far from what you are describing.

[sbierwagen]

The AlphaZero algorithm swapped between board games pretty easily. OpenAI could also have been gesturing at this when they named the GPT paper "Language Models are Few-Shot Learners".

[visarga]

DALL-E + CLIP models show a deep understanding of the relation between images and text.

[talor_a]

they mention in the demo video that the inspiration for codex came from GPT-3 users training it to respond to queries with code samples. I saw some pretty impressive demos of the original model creating SQL queries from plain questions. I'm not sure if that counts as switching domains, but it's something?

[simsla]

The problem with this (very popular) argument is that you can't give a CS course to a baby and expect them to get at programming.

By the time we see our first line of code, most of us have seen a ridiculous amount of data. We've been trained in problem solving, logical reasoning, maths, natural language processing, ... Hell, we've been trained as pattern matchers since we've been born.

By my account, humans actually need a large amount of training data. It might be the knowledge federation and generalisation that we're good at, but I don't think we're a clear winner in data efficiency.

[randallsquared]

Taking 11Mbps [1] as the raw uncompressed incoming data, and assuming 16 hours of waking environment consumption on average (likely high for children), a 13yo has taken in less than 400 TB of information (I used 11 * 60 * 60 * 16 * 365 * 13 / 8.) That's... surprisingly low.

[1] https://www.britannica.com/science/information-theory/Physio...

[naresh_xai]

Are we still limiting to visual cues and not the auditory,smell,taste,touch data which we get exposed to?

[FeepingCreature]

Visual input is so dense it's basically not worth tracking the other senses from a data rate pov.

[maxwells-daemon]

I would argue humans ingest a lot more than 159GB before they can write code. Most of it isn't Python, and humans currently transfer knowledge a lot more efficiently than NNs, but I suspect that'll change as incorporating more varied data sources becomes feasible.

[pnt12]

We generalize pretty well. One could say: "it took you 20 years to learn python!", but actually I learned python, Java, c#... Software engineering, machine learning... How to play guitar, how to cook.. .How to speak Portuguese, how to speak English... And thousands and thousands of different things which build on each other.

You can give a programmer a few kb of code in a new language and that will give him a small grasp of how it works.

[lstmemery]

I have to disagree with you here. In the Codex paper[1], they have two datasets that Codex got correct about 3% of the time. These are interview and code competition questions. From the paper:

"Indeed, a strong student who completes an introductory computer science course is expected to be able to solve a larger fraction of problems than Codex-12B."

This suggests to me that Codex really doesn't understand anything about the language beyond syntax. I have no doubt that future systems will improve on this benchmark, but they will likely take advantage of the AST and could use unit tests in a RL-like reward function.

[1] https://arxiv.org/abs/2107.03374

[ruuda]

> but they will likely take advantage of the AST

In the end, a more general approach with more compute, always wins over applying domain knowledge like taking advantage of the AST. This is called “the bitter lesson”. http://www.incompleteideas.net/IncIdeas/BitterLesson.html

[lstmemery]

I don't think the bitter lesson is applies to ASTs.

From the Bitter Lesson:

"Early methods conceived of vision as searching for edges, or generalized cylinders, or in terms of SIFT features. But today all this is discarded. Modern deep-learning neural networks use only the notions of convolution and certain kinds of invariances, and perform much better."

Those models are taking advantage of inductive biases. Every model has them, including the massive language models. They are not the same as engineered features (such as SIFTs) or heuristics.

Using the AST is just another way of looking at the code already in your dataset. For the model to understand what it is writing, it needs to map the text sequences map to ASTs anyways. It can attempt to learn this, but the 12B model still makes illegal Python code so it clearly hasn't.

[kevinqi]

"the bitter lesson" is a very interesting, thank you! However, I wonder if AST vs. text analysis is fully comparable to the examples given in the post. Applying human concepts for chess, go, image processing, etc. failed over statistical methods, but I don't think AST vs. text is exactly the same argument. IMO, using an AST is simply a more accurate representation of a program and doesn't necessarily imply an attempt to bring in human intuition/concepts.

[modeless]

I mean, the AST doesn't help at all with comments which are potentially the most valuable part of the code to an AI like this. Formatting is also ignored by the AST but may play a role in understanding, just as it can for humans.

The model can clearly already generate large amounts of code with no syntax errors in one shot. It's probably better at that than I am, I always need to fix something after typing a bunch of code without calling the compiler. I think that instead of adding a bunch of language-specific AST stuff it would be far better to simply give the model the ability to iterate on its solution the way humans do, to fix any syntax errors or logic bugs discovered by the compiler or at runtime. That could potentially work in a generic way for any language. It seems like the obvious next step, though figuring out how to train it is not obvious.

[nmca]

12B, though. What about 1.2T?

[lstmemery]

You need to scale the amount of data to take advantage of the increase in parameters. I'm not sure where we would find another 100 GitHubs worth of data.

[andreyk]

"I see no reason that this technology couldn't smoothly scale into human-level intelligence, yet lots of people seem to think it'll require a step change or is impossible."

I am a big fan of LMs and am not in the don't really understand crowd, but here are a couple of reasons:

1. Large language models such as GPT or Codex still have several major architectural limitations. They lack the ability to make use of long-term memory, since they have a fairly limited amount of info they can take as input; GPT 3 is great at short stories, but can't go beyond that, and it's hard to prime it with a lot of information as you would eg a new employee. There is some work on this, but afaik not very much and it's very much unsolved.

2. Large language models have only gotten this good by ingesting massive amounts of data and scaling up compute. Yet, this growth comes with diminish returns for every order of magnitude. So it just not being to scale either the data or the compute needs sufficiently (with existing hardware architectures) is a very plausible reason.

3. Large language models 'have it easy' because they only deal with one modality (text). Humans intelligence on the other hand is multimodal - we can process vision inputs, sound, touch, etc. sound, etc. simultaneously and share concepts between these modalities. And we likewise output motor commands that result in motion, text. So far it's not too obvious how to achieve this - OpenAI took a step with DALL-E, but that was by just mining a massive amount of image-text pairs, and it's not obvious this is easy for other modalities, in particular for motor control.

4. Human-level intelligence is often framed as having system 1 (reactive output) and system 2 (longer term reasoning not in response to immediate stimuli) - this is not at all present in language models.

5. related to above two, at least some of human intelligence is derived from reinforcement learning (optimizing a policy that is multi-step with a delayed reward). This is much harder than the plain self-supervised learning of LMs.

And probably there are a bunch more like these. So while I do think these sorts of models represent a lot of progress, there are many reasons to be doubtful that just 'scale it up' will work to get much further.

[maxwells-daemon]

As an add-on to this: I'd encourage anyone interested in this debate to read Rich Sutton's "The Bitter Lesson" (http://www.incompleteideas.net/IncIdeas/BitterLesson.html).

At every point in time, the best systems we can build today will be ones leveraging lots of domain-specific information. But the systems that will continue to be useful in five years will always be the ones freely that scale with increased parallel compute and data, which grow much faster than domain-specific knowledge. Learning systems with the ability to use context to develop domain-specific knowledge "on their own" are the only way to ride the wave of this computational bounty.

[pchiusano]

https://rodneybrooks.com/a-better-lesson/ is an interesting retort to the Sutton post.

[fpgaminer]

> "language models don't really understand anything"

I have a sneaking suspicion that, if blinded, the crowd of people saying variations of that quote would also identify the vast majority of human speech as regurgitated ideas as well.

> I see no reason that this technology couldn't smoothly scale into human-level intelligence

Yup, the OpenAI scaling paper makes this abundantly clear. There is currently no end in sight for the size that we can scale GPT to. We can literally just throw compute at the problem and GPT will get smarter. That's never been seen before in ML. Last time I ran the calculations I estimated that, everything else being equal, we'd reach GPT-human in 20 years (GPT with similar parameter scale as a human brain). That's everything else being equal. It is more than likely that in the next twenty years innovation will make GPT and the platforms we use to train and run models like it more efficient.

And the truly terrifying thing is that, to me, GPT-3 has about the intelligence of a bug. Yet it's a bug who's whole existence is human language. It doesn't have to dedicate brain power to spatial awareness, navigation, its body, handling sensory input, etc. GPT-human will be an intelligence with the size of a human brain, but who's sole purpose is understanding human language. And it's been to every library to read every book ever written. In every language. Whatever failings GPT may have at that point, it will be more than capable of compensating for in sheer parameter count, and leaning on the ability to combine ideas across the _entire_ human corpus.

All available through an API.

[raducu]

Sorry for my limited knowledge of GPT, but isn't it limited by the training data set, like all other models?

[bufferoverflow]

It probably can scale, but we're nowhere near the computational power we need to even recreate the brain. And don't forget, our brain took a billion years to evolve.

A typical brain has 80-90 billion neurons and 125 trillion synapses. That's a big freaking network to train.

Hopefully we can figure out how to train parts of it and then assemble something very smart.

[jacquesm]

Takes on average 2.5 decades to train it.

[mattkrause]

That's just from the most recent checkpoint :-)

If you were to build it "from scratch" you'd also need to include the millions of years of (distributed) evolution required to get that particular kid to that point.

Tony Zador has some interesting thoughts about that, including"A critique of pure learning", here: https://www.nature.com/articles/s41467-019-11786-6)

[sailingparrot]

The definition of "understanding" behaves just like the definition of "intelligence": The threshold to qualify gets pushed by as much as the technology progresses, so that nothing we create is ever intelligent and nothing ever understands.

[jdonaldson]

I think intelligence as defined as "mapping inputs into goal states" is pretty well handled by models, and the models may be able to pick and choose states that are sufficient for achieving the goals.

However, the intelligence that's created by language models is very schizophrenic, and the human-level reflective intelligence that it displays is at best a bit of Frankenstein's monster (an agglomeration of utterances from other people that it uses to form sentences that form opinions of itself or its world).

I think that modeling will help us learn more about human intelligence, but we're going to have to do a lot better than just training models blindly on huge amounts of text.

[visarga]

Maybe we're also >50% Frankenstein monsters, an agglomeration of utterances from other people.

[nxmnxm99]

Humans (technologists) in particular are awful at extrapolating. Transformers being able to combine rudimentary, defined "concepts" and scaling that into human intelligence makes about as much sense as extrapolating a XOR gate or an if-else statement "scaling" to human intelligence.

I'm of the "human beings are much more than big linear algebra functions slapped on top of a large processor" crowd.

[marto1]

To play devils's advocate: well, it might just very well turn out that what most humans are currently engaged in can very well turn out to be reducible to "big linear algebra functions slapped on top of a large processor".

And then the part about "being just like humans" will be the marketing gravy train that funds the operation.

[8eye]

i am exactly where you are, it’s not a matter of if, merely a matter of when