[ferguess_k]

Can you please explain "the transfer function is negative"?

I'm wondering whether one has tested with the same model but on two situations:

1) Bring it to superhuman level in game A and then present game B, which is similar to A, to it.

2) Present B to it without presenting A.

If 1) is not significantly better than 2) then maybe it is not carrying much "knowledge", or maybe we simply did not program it correctly.

[tough]

I think the problem is we train models to pattern match, not to learn or reason about world models

[singron]

I think this is clearly a case of over fitting and failure to generalize, which are really well understood concepts. We don't have to philosophize about what pattern matching really means.

[magicalhippo]

In the Physics of Language Models[1] they argue that you must augment your training data by changing sentences and such, in order for the model to be able to learn the knowledge. As I understand their argument, language models don't have a built-in way to detect what is important information and what is not, unlike us. Thus the training data must aid it by presenting important information in many different ways.

Doesn't seem unreasonable that the same holds in a gaming setting, that one should train on many variations of each level. Change the lengths of halls connecting rooms, change the appearance of each room, change power-up locations etc, and maybe even remove passages connecting rooms.

[1]: https://physics.allen-zhu.com/part-3-knowledge/part-3-1

[NBJack]

In other words, they learn the game, not how to play games.

[fsmv]

They memorize the answers not the process to arrive at answers

[EternalFury]

They learn the value of specific actions in specific contexts based on the rewards they received during their play time. Specific actions and specific contexts are not transferable for various reasons. John quoted that varying frame rates and variable latency between action and effect really confuse the models.

[nightpool]

Okay, so fuzz the frame rate and latency? That feels very easy to fix.

[wredcoll]

Good point, you should write to John Carmack and let him know you've figured out the problem.

[IshKebab]

This has been disproven so many times... They clearly do both. You can trivially prove this yourself.

[0xWTF]

> You can trivially prove this yourself.

Given the long list of dead philosophers of mind, if you have a trivial proof, would you mind providing a link?

[IshKebab]

Just go and ask ChatGPT or Claude something that can't possibly be in its training set. Make something up. If it is only memorising answers then it will be impossible for it to get the correct result.

A simple nonsense programming task would suffice. For example "write a Python function to erase every character from a string unless either of its adjacent characters are also adjacent to it in the alphabet. The string only contains lowercase a-z"

That task isn't anywhere in its training set so they can't memorise the answer. But I bet ChatGPT and Claude can still do it.

Honestly this is sooooo obvious to anyone that has used these tools, it's really insane that people are still parroting (heh) the "it just memorises" line.

[pdabbadabba]

It’s really easy: go to Claude and ask it a novel question. It will generally reason its way to a perfectly good answer even if there is no direct example of it in the training data.

[beefnugs]

yeahhhh why isnt there a training structure where you play 5000 games, and the reward function is based on doing well in all of them?

I guess its a totaly different level of control: instead of immediately choosing a certain button to press, you need to set longer term goals. "press whatever sequence over this time i need to do to end up closer to this result"

There is some kind of nested multidimensional thing to train on here instead of immediate limited choices

[IshKebab]

Well yeah... If you only ever played one game in your life you would probably be pretty shit at other games too. This does not seem very revealing to me.

[trainerxr50]

I am decent at chess but barely know how the pieces in Go move.

Of course, this because I have spent a lot of time TRAINING to play chess and basically none training to play go.

I am good on guitar because I started training young but can't play the flute or piano to save my life.

Most complicated skills have basically no transfer or carry over other than knowing how to train on a new skill.

[drw85]

But the point here is, if i gave you a guitar with a string more or less. Or a different shaped guitar, you could play it.

If i give you a chess set with dwarf themed pieces and different colored squares, you could play immediately.

[e2021]

I don't think thats true. If you'd only ever played Doom, I think you could play, say, counterstrike or half-life and be pretty good at it, and i think Carmack is right that its pretty interesting that this doesn't seem to be the case for ai models

[antisthenes]

Where do you draw the line between pattern matching and reasoning about world models?

A lot of intelligence is just pattern matching and being quick about it.

[halfcat]

The line is: building an internal world model requires interfacing with the world, not a model of it, and subsequent failing (including death and survivorship over generations) and adaptation. Plus pattern matching.

Current AI only does one of those (pattern matching, not evolution), and the prospects of simulating evolution is kind of bleak, given I don’t think we can simulate a full living cell yet from scratch? Building a world model requires life (or something that has undergone a similar evolutionary survivorship path), not something that mimics life.

[BlueTemplar]

You don't need to simulate a full living cell to have evolution. In fact, isn't using evolving programs a decades-old technic ?

[halfcat]

Genetic programming models a natural process of evolution to do something useful, the same way machine learning models neurons to do something useful.

But producing something useful is a totally different thing from producing resilience in physical reality. That takes a world model, and I guess my suspicion is that an entity can’t build a world model without a long history of surviving in that world.

Put another way, you can never replicate what it’s like to burn your hand on the fire using only words. You could have a million people tell a child about what fire is like, the dangers of it, the power of it, the pain of it. But they will never develop an innate understanding of it that helps them navigate the real world.

Until they stick their hand in the fire. Then they know.

[ferguess_k]

I kinda think I'm more or less the same...OK maybe we have different definitions of "pattern matching".

[veqz]

It's Plato's cave:

We train the models on what are basically shadows, and they learn how to pattern match the shadows.

But the shadows are only depictions of the real world, and the LLMs never learn about that.

[ebonnafoux]

But the same is true for human, we get our information though our senses we do not have the __real__ word directly.

[actionfromafar]

We do much more than LLMs have. We have bodies and feelings.

[EternalFury]

100%

[Zanfa]

According to Carmack's recent talk [0], SOTA models that have been trained on game A don't perform better or train faster on game B. Even worse, training on game B negatively affects performance in game A when returning to it.

[0] https://www.youtube.com/watch?v=3pdlTMdo7pY

[rhdunn]

You can see a similar effect with LLM finetunes. If you finetune a base model (or other instruct/finetune model) for a new task (e.g. better maths or programming language comprehension) it performs worse at other tasks like creative writing.

To mitigate this you have to include the other categories in your finetune training dataset so it doesn't lose the existing knowledge. Otherwise, the backpropagation and training will favour weights that reflect the new data.

In the game example having the weights optimized for game A doesn't help with game B. It would be interesting to see if training for both game A and B help it understand concepts in both.

Similarly with programming languages it would be interesting to see if training it with multiple languages if it can extract concepts like if statements and while loops.

IIUC from the observations with multilingual LLMs you need to have the different things you are supporting in the training set together. Then the current approach is able to identify similar concepts/patterns. It's not really learning these concepts but is learning that certain words often go together or that a word in one language is similar to another.

It would be interesting to study multilingual LLMs for their understanding of those languages in the case where the two languages are similar (e.g. Scottish and Irish Gaelic; Dutch and Afrikaans; etc.), are in the same language family (French, Spanish, Portuguese), or are in different language families (Italian, Japanese, Swahili), etc.

[Zanfa]

> In the game example having the weights optimized for game A doesn't help with game B. It would be interesting to see if training for both game A and B help it understand concepts in both.

Supposedly it does both A and B worse. That's their problem statement essentially. Current SOTA models don't behave like humans would. If you took a human that's really good at A and B, chances are they're gonna pick up C much quicker than a random person off the street that hasn't even seen Atari before. With SOTA models, the random "person" does better at C than the A/B master.