[SomeCallMeTim]

LLMs are good at tasks that don't require actual understanding of the topic.

They can come up with excellent (or excellent-looking-but-wrong) answers to any question that their training corpus covers. In a gross oversimplification, the "reasoning" they do is really just parroting a weighted average (with randomness injected) of the matching training data.

What they're doing doesn't really match any definition of "understanding." An LLM (and any current AI) doesn't "understand" anything; it's effectively no more than a really big, really complicated spreadsheet. And no matter how complicated a spreadsheet gets, it's never going to understand anything.

Not until we find the secret to actual learning. And increasingly it looks like actual learning probably relies on some of the quantum phenomena that are known to be present in the brain.

We may not even have the science yet to understand how the brain learns. But I have become convinced that we're not going to find a way for digital-logic-based computers to bridge that gap.

[jwells89]

This is also why image generating models struggle to correctly draw highly variable objects like limbs and digits.

They’ll be able to produce infinite good looking cardboard boxes, because those are simple enough to be represented reasonably well with averages of training data. Limbs and digits on the other hand have nearly limitless different configurations and as such require an actual understanding (along with basic principles such as foreshortening and kinetics) to be able to draw well without human guidance.

[grobgambit]

I would just add that I think I have encountered situations that knowing the weighted average answer from the training data for topics I didn't previously understand created better initial conditions for MY learning of the topic than not knowing the weighted average answer.

The problem to me is we are holding LLMs to a standard of usefulness from science fiction and not reality.

A new, giant set of encyclopedias has enormous utility but we wouldn't hold it against the encyclopedias that they aren't doing the thinking for us or 100% omniscient.

[naasking]

> What they're doing doesn't really match any definition of "understanding."

What is the mechanistic definition of "understanding"?

[throwthrowuknow]

What is your definition of understanding?

Please show me where the training data exists in the model to perform this lookup operation you’re supposing. If it’s that easy I’m sure you could reimplement it with a simple vector database.

Your last two paragraphs are just dualism in disguise.

[Etherlord87]

I'm far from being an expert on AI models, but it seems you lack the basic understanding of how these models work. They transform data EXACTLY like spreadsheets do. You can implement those models in Excel, assuming there's no row or column limit (or that it's high enough) - of course it will be much slower than the real implementations, but OP is right - LLMs are basically spreadsheets.

Question is, wouldn't a brain qualify as a spreadsheet, do we know it can't be implemented as one? Well, maybe not, I'm not an expert on spreadsheets either, but I think spreadsheets don't allow you circular references, and brain does, you can have feedback loops in the brain. So even if the brain doesn't have something still not understood by us, that OP suggests, it still is more powerful than AI.

BTW, this is one explanation on why AI fails at some tasks: ask AI if two words rhyme and it will be quite reliable on that. But ask it to give you word pairs that rhyme, and it will fail, because it won't run an internal loop trying some words and checking if they succeed to rhyme or not. If some AI actually succeeds at rhyming, it would do so either because it's trained to contain such word pairs from the get-go or because it's implemented to have multiple passes or something...

[throwthrowuknow]

You can implement Doom in a spreadsheet too, so what? That wasn’t the point op or I were making. If you bother to read the sentence before op talks about spreadsheets they are making the conjecture that LLMs are lookup tables operating on the corpus they were trained on. That is the aspect of spreadsheets they were comparing them to, not the fact that spreadsheets can be used to implement anything that any other programming language can. Might as well say they are basically just arrays with some functions in between, yeah no shit.

Which LLMs can’t produce rhyming pairs? Both the current ChatGPT 3.5 and 4 seem to be able to generate as many as I ask for. Was this a failure mode at some point?

[lossolo]

> Which LLMs can’t produce rhyming pairs? Both the current ChatGPT 3.5 and 4 seem to be able to generate as many as I ask for

Only in english. If they would understand language and rhymes they would do it in every other language it knows, It can't in my language while it can speak in it fluently. It just fails. And fails in so many other areas, I'm using LLMs daily for work and other stuff and if you use them long enough you will see that they are statistical machines not intelligent entities.

[singron]

People are confusing the limited computational model of a transformer with the "Chinese room argument", which leads to unproductive simultaneous debates of computational theory and philosophy.

[SomeCallMeTim]

I'm not confusing anything. I'm familiar with the Chinese Room Argument and I know how LLMs work.

What I'm saying is arguably philosophically related, in that I'm saying the LLM's model is analogous to the "response book" in the room. It doesn't matter how big the book is; if the book never changes, then no learning can happen. If no learning can happen, then understanding, a process that necessarily involves active reflection on a topic, can exist.

You simply can't say a book "understands" anything. To understand is to contemplate and mentally model a topic to the point where you can simulate it, at least at a high level. It's dynamic.

An LLM is static. It can simulate a dynamic response by having multiple stages that dig through an multiple insanely large books of instructions that cross reference each other and that involve calculations and bookmarks and such to come up with a result--but the books never change as part of the conversation.

[lossolo]

Transformer is not a simple vector database doing simple lookup operation. It's doing lookup operation on a pattern, not a word. It learns patterns from the dataset. If your pattern is not there it will hallucinate or give you the wrong answer like GPT4 and Opus gave me hundreds of times already.

[tibbydudeza]

>> quantum phenomena

You mean like the microtubles of Roger Penrose ???.

https://www.youtube.com/watch?v=jG0OpvudA10

[danenania]

> the "reasoning" they do is really just parroting a weighted average (with randomness injected) of the matching training data

Perhaps our brains are doing exactly the same, just with more sophistication?

[SomeCallMeTim]

No.

We know how current deep learning neural networks are trained.

We know definitively that this is not how brains learn.

Understanding requires learning. Dynamic learning. In order to experience something, an entity needs to be able to form new memories dynamically.

This does not happen anywhere in current tech. It's faked in some cases, but no, it doesn't really happen.

[danenania]

> We know definitively that this is not how brains learn.

Ok then, I guess the case is closed.

> an entity needs to be able to form new memories dynamically.

LLMs can form new memories dynamically. Just pop some new data into the context.

[SomeCallMeTim]

> LLMs can form new memories dynamically. Just pop some new data into the context.

No, that's an illusion.

The LLM itself is static. The recurrent connections form a soft-of temporary memory that doesn't affect the learned behavior of the network at all.

I don't get why people who don't understand what's happening keep arguing that AIs are some sci-fi interpretation of AI. They're not. At least not yet.

[danenania]

It isn't temporary if you keep it permanently in context (or in a RAG store) and pass it into every model call, which is how long-term memory is being implemented both in research and in practice. And yes it obviously does affect the learned behavior. The distinction you're making between training and context is arbitrary.

[naasking]

> We know definitively that this is not how brains learn.

So you have mechanistic, formal model of how the brain functions? That's news to me.

[Scarblac]

Your brain was first trained by reading all of the Internet?

Anyway, the question of whether computers can think is as interesting as the question whether submarines can swim.

[naasking]

> Anyway, the question of whether computers can think is as interesting as the question whether submarines can swim.

Given the amount of ink spilled on the question, gotta disagree with you there.

[Scarblac]

For the record, that wasn't me, it's a famous quote from Edsger Dijkstra.

[iraqmtpizza]

Endless ink has been spilled on the most banal and useless things. Deconstructing ice cream and physical beauty from a Marxist-feminist race-conscious postmodern perspective.

[SomeCallMeTim]

There's no way brains have the "right answers" fed into them as required by backpropagation.

[naasking]

Look up predictive coding. Our senses are constantly feeding us corrections to our predictions.

[xanderlewis]

Every single discussion of ‘AGI’ has endless comments exactly like this. Whatever criticism is made of an attempt to produce a reasoning machine, there’s always inevitably someone who says ‘but that’s just what our brains do, duhhh… stop trying to feel special’.

It’s boring, and it’s also completely content-free. This particular instance doesn’t even make sense: how can it be exactly the same, yet more sophisticated?

Sorry.

[adrianN]

The problem is that we currently lack good definitions for crucial words such as "understanding" and we don't know how brains work, so that nobody can objectively tell whether a spreadsheet "understands" anything better than our brains. That makes these kinds of discussions quite unproductive.

[xanderlewis]

I can’t define ‘understanding’ but I can certainly identify a lack of it when I see it. And LLM chatbots absolutely do not show signs of understanding. They do fine at reproducing and remixing things they’ve ‘seen’ millions of times before, but try asking them technical questions that involve logical deduction or an actual ability to do on-the-spot ‘thinking’ about new ideas. They fail miserably. ChatGPT is a smooth-talking swindler.

I suspect those who can’t see this either

(a) are software engineers amazed that a chatbot can write code, despite it having been trained on an unimaginably massive (morally ambiguously procured) dataset that probably already contains something close to the boilerplate you want anyway

(b) don’t have the sufficient level of technical knowledge to ask probing enough questions to betray the weaknesses. That is, anything you might ask is either so open-ended that almost anything coherent will look like a valid answer (this is most questions you could ask, outside of seriously technical fields) or has already been asked countless times before and is explicitly part of the training data.

[danenania]

Your understanding of how LLMs work isn’t at all accurate. There’s a valid debate to be had here, but it requires that both sides have a basic understanding of the subject matter.

[xanderlewis]

How is it not accurate? I haven’t said anything about the internal workings of an LLM — just what it able to produce (which is based on observation).

I have more than a basic understanding of the subject matter (neural networks; specifically transformers, etc.). It’s actually not a hugely technical field.

By the way, it appears that you are in category (a).

[danenania]

As the comment I replied to very correctly said, we don’t know how the brain produces cognition. So you certainly cannot discard the hypothesis that it works through “parroting” a weighted average of training data just as LLMs are alleged to do.

Considering that LLMs with a much smaller number of neurons than the brain are in many cases producing human-level output, there is some evidence, if circumstantial, that our brains may be doing something similar.

[iraqmtpizza]

LLMs don't have neurons. That's just marketing lol.

"A neuron in a neural network typically evaluates a sequence of tokens in one go, considering them as a whole input." -- ChatGPT

You could consider an RTX 4090 to be one neuron too.

[danenania]

It’s almost as if ‘neuron’ has a different meaning in computer science than biology.

[iraqmtpizza]

LOL you just owned the guy who said "LLMs with a much smaller number of neurons than the brain are in many cases producing human-level output"

[xanderlewis]

> in many cases producing human-level output

They’re not, unless you blindly believe OpenAI press releases and crypto scammer AI hype bros on Twitter.