[EternalFury]

What John Carmack is exploring is pretty revealing. Train models to play 2D video games to a superhuman level, then ask them to play a level they have not seen before or another 2D video game they have not seen before. The transfer function is negative. So, in my definition, no intelligence has been developed, only expertise in a narrow set of tasks.

It’s apparently much easier to scare the masses with visions of ASI, than to build a general intelligence that can pick up a new 2D video game faster than a human being.

[ozgrakkurt]

Seeing comments here saying “this problem is already solved”, “he is just bad at this” etc. feels bad. He has given a long time to this problem by now. He is trying to solve this to advance the field. And needless to say, he is a legend in computer engineering or w/e you call it.

It should be required to point to the “solution” and maybe how it works to say “he just sucks” or “this was solved before”.

IMO the problem with current models is that they don’t learn categorically like: lions are animals, animals are alive. goats are animals, goats are alive too. So if lions have some property like breathing and goats also have it, it is likely that other similar things have the same property.

Or when playing a game, a human can come up with a strategy like: I’ll level this ability and lean on it for starting, then I’ll level this other ability that takes more time to ramp up while using the first one, then change to this play style after I have the new ability ready. This might be formulated completely based on theoretical ideas about the game, and modified as the player gets more experience.

With current AI models as far as I can understand, it will see the whole game as an optimization problem and try to find something at random that makes it win more. This is not as scalable as combining theory and experience in the way that humans do. For example a human is innately capable of understanding there is a concept of early game, and the gains made in early game can compound and generate a large lead. This is pattern matching as well but it is on a higher level .

Theory makes learning more scalable compared to just trying everything and seeing what works

[93po]

I'm a huge fan of Carmack and read the book (Masters of Doom) multiple times and love it, too. But he's a legend for pioneering PC gaming graphics in a way that was feasible for a single (very talented) person to accomplish, and was also pioneering something that already existed on consoles. I think there's a big leap from very cleverly recreating existing very basic and simple 3d graphics for a new platform versus the massive task that is AGI/ASI, which I don't think is something a single person can meaningfully move forward at this point. Even the big jump we got from GPTs was due to many many people.

[Olreich]

Current models are lossy databases at this point. Carmack looks like he might be trying to get logical reasoning to work (learning something abstract in one context and applying it to a similar context). That is something that would advance the field significantly and may be possible with a small team of researchers.

[motorest]

> Seeing comments here saying “this problem is already solved”, “he is just bad at this” etc. feels bad. He has given a long time to this problem by now. He is trying to solve this to advance the field. And needless to say, he is a legend in computer engineering or w/e you call it.

This comment, with the exception of the random claim of "he is just bad at this", reads like a thinly veiled appeal to authority. I mean, you're complaining about people pointing out prior work, reviewing the approach, and benchmarking the output.

I'm not sure you are aware, but those items (bibliographical review, problem statement, proposal, comparison/benchmarks) are the very basic structure of an academic paper, which each and every single academic paper on any technical subject are required to present in order to be publishable.

I get that there must be a positive feedback element to it, but pay attention to your own claim: "He is trying to solve this to advance the field." How can you tell whether this really advances the field if you want to shield it from any review or comparison? Otherwise what's the point? To go on and claim that ${RANDOM_CELEB} parachuted into a field and succeeded at first try where all so-called researchers and experts failed?

Lastly, "he is just bad at this". You know who is bad at research topics? Researchers specialized on said topic. Their job is to literally figure out something they don't know. Why do you think someone who just started is any different?

[vladimirralev]

He is not using appropriate models for this conclusion and neither is he using state of the art models in this research and moreover he doesn't have an expensive foundational model to build upon for 2d games. It's just a fun project.

A serious attempt at video/vision would involve some probabilistic latent space that can be noised in ways that make sense for games in general. I think veo3 proves that ai can generalize 2d and even 3d games, generating a video under prompt constraints is basically playing a game. I think you could prompt veo3 to play any game for a few seconds and it will generally make sense even though it is not fine tuned.

[sigmoid10]

Veo3's world model is still pretty limited. That becomes obvious very fast once you prompt out of distribution video content (i.e. stuff that you are unlikely to find on youtube). It's extremely good at creating photorealistic surfaces and lighting. It even has some reasonably solid understanding of fluid dynamics for simulating water. But for complex human behaviour (in particular certain motions) it simply lacks the training data. Although that's not really a fault of the model and I'm pretty sure there will be a way to overcome this as well. Maybe some kind of physics based simulation as supplement training data.

[mym1990]

What is the basis for it having a reasonable understanding of fluid dynamics? Why don’t you think it’s just regurgitating some water scenes derived from its training data, rather than generating actual fluid dynamics?

[sigmoid10]

Because it can actually extrapolate to unseen cases while maintaining realism.

[mym1990]

Ah yes, the classic “because it can” argument. I’ll take that to mean you don’t know what you’re talking about.

[sigmoid10]

It seems you are confusing this with a personal opinion. This is not my opinion. This is merely the consensus of current research.

See here for example:

[1] https://arxiv.org/pdf/2410.18072

[2] https://arxiv.org/pdf/2411.02914v1

[3] https://openai.com/index/video-generation-models-as-world-si...

But even if you knew nothing about this topic, the observation that you simply couldn't store the necessary amount of video data in a model such that it could simply regurgitate it should give you a big clue as to what is happening.

[altairprime]

Is any model currently known to succeed in the scenario that Carmack’s inappropriate model failed?

[outofpaper]

No monolithic models but us ng hybrid approaches we've been able to beet humans for some time now.

[altairprime]

To confirm: hybrid approaches can demonstrate competence at newly-created video games within a short period of exposure, so long as similar game mechanics from other games were incorporated into their training set?

[317070]

What you're thinking of is much more like the Genie model from DeepMind [0]. That one is like Veo, but interactive (but not publically available)

[0] https://deepmind.google/discover/blog/genie-2-a-large-scale-...

[Intralexical]

> I think veo3 proves that ai can generalize 2d and even 3d games, generating a video under prompt constraints is basically playing a game.

In the same way that keeping a dream journal is basically doing investigative journalism, or talking to yourself is equivalent to making new friends, maybe.

The difference is that while they may both produce similar, "plausible" output, one does so as a result of processes that exist in relation to an external reality.

[troupo]

> I think veo3 proves that ai can generalize 2d and even 3d games

It doesn't. And you said it yourself:

> generating a video under prompt constraints is basically playing a game.

No. It's neither generating a game (that people can play) nor is it playing a game (it's generating a video).

Since it's not a model of the world in any sense of the word, there are issues with even the most basic object permanenece. E.g. here's veo3 generating a GTA-style video. Oh look, the car spins 360 and ends up on a completely different street than the one it was driving down previously: https://www.youtube.com/watch?v=ja2PVllZcsI

[vladimirralev]

It is still doing a great job for a few frames, you could keep it more anchored to the state of the game if you prompt it. Much like you can prompt coding agents to keep a log of all decisions previously made. Permanenece is excellent, it slips often but it mostly because it is not grounded to specific game state by the prompt or by the decision log.

[troupo]

So, "it generates a game" somehow "it's incapable of maintaining basic persistence without continuous prompting per frame".

Also, prompting doesn't work as you imply it does.

[keerthiko]

> generating a video under prompt constraints is basically playing a game

Besides static puzzles (like a maze or jigsaw) I don't believe this analogy holds? A model working with prompt constraints that aren't evolving or being added over the course of "navigating" the generation of the model's output means it needs to process 0 new information that it didn't come up with itself — playing a game is different from other generation because it's primarily about reacting to input you didn't know the precise timing/spatial details of, but can learn that they come within a known set of higher order rules. Obviously the more finite/deterministic/predictably probabilistic the video game's solution space, the more it can be inferred from the initial state, aka reduce to the same type of problem as generating a video from a prompt), which is why models are still able to play video games. But as GP pointed out, transfer function negative in such cases — the overarching rules are not predictable enough across disparate genres.

> I think you could prompt veo3 to play any game for a few seconds

I'm curious what your threshold for what constitutes "play any game" is in this claim? If I wrote a script that maps button combinations to average pixel color of a portion of the screen buffer, by what metric(s) would veo3 be "playing" the game more or better than that script "for a few seconds"?

edit: removing knee-jerk reaction language

[vladimirralev]

It's not ideal, but you can prompt it with an image of a game frame, explain the objects and physics in text and let it generate a few frames of gameplay as a substitute for controller input as well as what it expects as an outcome. I am not talking about real interactive gameplay.

I am just saying we have proof that it can understand complex worlds and sets of rules, and then abide by them. It doesn't know how to use a controller and it doesn't know how to explore the game physics on its own, but those steps are much easier to implement based on how coding agents are able to iterate and explore solutions.

[pshc]

I think we need a spatial/physics model handling movement and tactics watched over by a high level strategy model (maybe an LLM).

[IIAOPSW]

There's something fascinating about this, because the human ability to "transfer knowledge" (eg pick up some other never before seen video game and quickly understand it) isn't really that general. There's a very particular "overtone window" of the sort of degrees of difference where it is possible.

If I were to hand you a version of a 2d platformer (lets say Mario) where the gimmick is that you're actually playing the fourier transform of the normal game, it would be hopeless. You might not ever catch on that the images on screen are completely isomorphic to a game you're quite familiar with and possibly even good at.

But some range of spatial transform gimmicks are cleanly intuitive. We've seen this with games like vvvvvv and braid.

So the general rule seems to be that intelligence is transferable to situations that are isomorphic up to certain "natural" transforms, but not to "matching any possible embedding of the same game in a different representation".

Our failure to produce anything more than hyper-specialists forces us to question exactly is meant by the ability to generalize other than just "mimicking an ability humans seem to have".

[Certhas]

When studying physics, people eventually learn about Fourier transform, and they learn about quantum mechanics, where the Fourier transform switches between describing things in terms of position and of momentum. And amazingly the harmonic oscillator is the same in position and momentum space! So maybe there are other creatures that perceive in momentum space! Everything is relative!

Except that's of course superficial nonsense. Position space isn't an accident of evolution, one of many possible encodings of spatial data. It's an extremely special encoding: The physical laws are local in position and space. What happens on the moon does not impact what happens when I eat breakfast much. But points arbitrarily far in momentum space do interact. Locality of action is a very very deep physical principle, and it's absolutely central to our ability to reason about the world at all. To break it apart into independent pieces.

So I strongly reject your example. It makes no sense to present the pictures of a video game in Fourier space. Its highly unnatural for very profound reasons. Our difficulty stems entirely from the fact that our vision system is built for interpreting a world with local rules and laws.

I also see no reason that an AI could successfully transfer between the two representations easily. If you start from scratch it could train on the Fourier space data, but that's more akin to using different eyes, rather than transfer.

[IIAOPSW]

But, you're not really rejecting my example, you're proving it. The human ability to generalize the concept of a 2d platformer is limited to a very narrow range of "intuitive" generalizations that have deeply baked assumptions in them like "locality of action". So when we try to replicate the ability to "generalize", at some point we have to recognize that we can't "generalize in general" but rather we have to deeply bake in certain assumptions about what sorts of variations on the learned theme are possible. Mario with some sort of gimmick that still respects locality of action is doable, the fourier transform of Mario isn't.

This is a problem because we are approaching AI from an angle of no a priori assumptions about the variations on the pattern that it should be able to generalize to. We just imagine that there's some magic way to recognize any isomorphic representation and transfer our knowledge to the new variables, when the reality is we can only recognize when the domain being transferred to is only different in a narrow set of ways like being upside down or on a bent surface. The set of possible variations on a 2d platformer we can generalize well enough to just pick up and play is a tiny subset of all the ways you could map the pixels on the screen to something else without technically losing information.

We could probably make an AI that bakes in the sort of assumptions where it can easily generalize what it learns to fourier space representations of the same data, but then it probably wouldn't be good at generalizing the same sorts of things we are good at generalizing.

My point (hypothesis really) is that the ability to "generalize in general" is a fiction. We can't do it either. But the sort of things we can generalize are exactly the sort that tend to occur in nature anyway so we don't notice the blind spot in what we can't do because it never comes up.

[chongli]

One of my favourite examples of games that are hard to train an AI on is The Legend of Zelda for NES. Many other games of the NES era have (at least in the short term) a goal function which almost perfectly corresponds to some simple memory value such as score or x-position.

Not Zelda. That game is highly nonlinear and its measurable goals (triforce pieces) are long-term objectives that take a lot of gameplay to obtain. As far as I’m aware, no AI has been able to make even modest progress without any prior knowledge of the game itself.

Yet many humans can successfully play and complete the first dungeon without any outside help. While completing the full game is a challenge that takes dedication, many people achieved it long before having access to the internet and its spoiler resources.

So why is this? Why are humans so much better at Zelda than AIs? I believe that transfer knowledge has a lot to do with it. For starters, Link is approximately human (technically Hylian, but they are considered a race of humans, not a separate species) which means his method of sensing and interacting with his world will be instantly familiar to humans. He’s not at all like an earthworm or an insect in that regard.

Secondly, many of the objects Link interacts with are familiar to most modern humans today: swords, shields, keys, arrows, money, bombs, boomerangs, a ladder, a raft, a letter, a bottle of medicine, etc. Since these objects in-game have real world analogues, players will already understand their function without having to figure it out. Even the triforce itself functions similarly to a jigsaw puzzle, making it obvious what the player’s final objective should be. Furthermore, many players would be familiar with the tropes of heroic myths from many cultures which the Zelda plot closely adheres to (undertake a quest of personal growth, defeat the nemesis, rescue the princess).

All of this cultural knowledge is something we take for granted when we sit down to play Zelda for the first time. We’re able to transfer it to the game without any effort whatsoever, something I have yet to witness an AI achieve (train an AI on a general cultural corpus containing all of the background cultural information above and get it to transfer that knowledge into gameplay as effectively as an unspoiled Zelda beginner).

As for the Fourier transform, I don’t know. I do know that the Legend of Zelda has been successfully completed while playing entirely blindfolded. Of course, this wasn’t with Fourier transformed sound, though since the blindfolded run relies on sound cues I imagine a player could adjust to the Fourier transformed sound effects.

[YokoZar]

I wonder if this is a case of overfitting from allowing the model to grow too large, and if you might cajole it into learning more generic heuristics by putting some constraints on it.

It sounds like the "best" AI without constraint would just be something like a replay of a record speedrun rather than a smaller set of heuristics of getting through a game, though the latter is clearly much more important with unseen content.

[justanotherjoe]

I don't get why people are so invested in framing it this way. I'm sure there are ways to do the stated objective. John Carmack isn't even an AI guy why is he suddenly the standard.

[GuB-42]

Who is an "AI guy"? The field as we know it is fairly new. Sure, neural nets are old hat, but a lot has happened in the last few years.

John Carmack founded Keen technology in 2022 and has been working seriously on AI since 2019. From his experience in the video game industry, he knows a thing or two about linear algebra and GPUs, that is the underlying maths and the underlying hardware.

So, for all intent and purposes, he is an "AI guy" now.

[amelius]

But the logic seems flawed.

He has built an AI system that fails to do X.

That does not mean there isn't an AI system that can do X. Especially considering that a lot is happening in AI, as you say.

Anyway, Carmack knows a lot about optimizing computations on modern hardware. In practice, that happens to be also necessary for AI. However, it is not __sufficient__ for AI.

[gerdesj]

"He has built an AI system that fails to do X."

Perhaps you have put your finger on the fatal flaw ...

[PeeMcGee]

> That does not mean there isn't an AI system that can do X.

You are holding the burden of proof here...

[amelius]

No, Carmack holds the burden of proof because he started the argument. His incapable program does not prove anything.

Maybe this is formulated a bit harshly, but let us respect the logic here.

[jeffreygoesto]

One of my supervisors used to say: "Don't tell me it's impossible, tell me _you_ could not do it." A true c_nt move that ends every discussion.

[amelius]

Huh, by saying that something is impossible, __you__ are ending the discussion, not your professor.

[lucumo]

No. Pointing out a flaw in an argument doesn't require proving the opposite.

[nkmnz]

This is exactly how Science works. He’s right until proven wrong. And so are you.

[qaq]

Keen includes researchers like Richard Sutton, Joseph Modayil etc. Also John has being doing it full time for almost 5 years now so given his background and aptitude for learning I would imaging by this time he is more of an AI guy then a fairly large percentage of AI PhDs.

[justanotherjoe]

Yeah and in another 5 years he'd probably be at nobel laureate level in AI. I don't think that's how it works. What do you mean? Even a phd program can take 5 years sometimes. Also the man started saying he'd bring about AGI right at the gate. He wasn't being exactly humble.

God I hate sounding like this. I swear I'm not too good for John Carmack, as he's infinitely smarter than me. But I just find it a bit weird.

I'm not against his discovery, just against the vibe and framing of the op.

[qaq]

He stated AGI is an interesting problem to work on could you provide a reference on him claiming "he'd bring about AGI right at the gate"?

[justanotherjoe]

Isn't that basically saying the same thing? I meant at the gate as he's speaking of AGI before the 5 years you mentioned

[varjag]

What in your opinion constitutes an AI guy?

[refulgentis]

Names >> all, and increasingly so.

One phenomena that bared this to me, in a substantive way, was noticing an increasing # of reverent comments re: Geohot in odd places here, that are just as quickly replied to by people with a sense of how he works, as opposed to the keywords he associates himself with. But that only happens here AFAIK.

Yapping, or, inducing people to yap about me, unfortunately, is much more salient to my expected mindshare than the work I do.

It's getting claustrophobic intellectually, as a result.

Example from the last week is the phrase "context engineering" - Shopify CEO says he likes it better than prompt engineering, Karpathy QTs to affirm, SimonW writes it up as fait accompli. Now I have to rework my site to not use "prompt engineering" and have a Take™ on "context engineering". Because of a couple tweets + a blog reverberating over 2-3 days.

Nothing against Carmack, or anyone else named, at all. i.e. in the context engineering case, they're just sharing their thoughts in realtime. (i.e. I don't wanna get rolled up into a downvote brigade because it seems like I'm affirming the loose assertion Carmack is "not an AI guy", or, that it seems I'm criticizing anyone's conduct at all)

EDIT: The context engineering example was not in reference to another post at the time of writing, now one is the top of front page.

[dvfjsdhgfv]

> Now I have to rework my site to not use "prompt engineering" and have a Take™ on "context engineering". Because of a couple tweets + a blog reverberating over 2-3 days.

The difference here is that your example shows a trivial statement and a change period of 3 days, whereas what Carmack is doing is taking years.

[refulgentis]

Right. Nothing against Carmack. Grew up on the guy. I haven't looked into, at all, into any of the disputed stuff, and should actively proclaim I'm a yuge Carmack fanboy.

[energy123]

Credentialism is bad, especially when used as a stick

[m_rpn]

Maybe cause he's like top 5 most influential computer programmers of all time and knew to be a super human workaholic?

[raincole]

Because it "confirms" what they already believe in.

[surecoocoocoo]

Ah some No True Scotsman

Not sure why justanotherjoe is a credible resource on who is and isn’t expert in some new dialectic and euphemism for machine state management. You’re that nobody to me :shrug:

Yann LeCun is an AI guy and has simplified it as “not much more than physical statistics.”

WWhole lot of AI is decades old info theory books applied to modern computer.

Either a mem value is or isn’t what’s expected. Either an entire matrix of values is or isn’t what’s expected. Store the results of some such rules. There’s your model.

The words are made up and arbitrary because human existence is arbitrary. You’re being sold on a bridge to nowhere.

[justanotherjoe]

I'm not being gatekeeper here. John Carmack came into AI around 2021 iirc and came in Lex Friedman and said he's going to bring about AGI. It's okay for him to try so but he had no particular expertise in the field. He's a brilliant guy and I'm not gonna say he's not going to succeed, or that his opinion is worthless. But people seeemed to think that the whole field is a farce just waiting for an adult to come in and fix it. I find that biased. By the way this is how people end up worshipping figures like Musk. There's a limit to transfer function of human expertise, ironically to the discussion at hand.

That's just what I think anyway.

[smokel]

The subject you are referring to is most likely Meta-Reinforcement Learning [1]. It is great that John Carmack is looking into this, but it is not a new field of research.

[1] https://instadeep.com/2021/10/a-simple-introduction-to-meta-...

[Uehreka]

These questions of whether the model is “really intelligent” or whatever might be of interest to academics theorizing about AGI, but to the vast swaths of people getting useful stuff out of LLMs, it doesn’t really matter. We don’t care if the current path leads to AGI. If the line stopped at Claude 4 I’d still keep using it.

And like I get it, it’s fun to complain about the obnoxious and irrational AGI people. But the discussion about how people are using these things in their everyday lives is way more interesting.

[ferguess_k]

Can you please explain "the transfer function is negative"?

I'm wondering whether one has tested with the same model but on two situations:

1) Bring it to superhuman level in game A and then present game B, which is similar to A, to it.

2) Present B to it without presenting A.

If 1) is not significantly better than 2) then maybe it is not carrying much "knowledge", or maybe we simply did not program it correctly.

[tough]

I think the problem is we train models to pattern match, not to learn or reason about world models

[singron]

I think this is clearly a case of over fitting and failure to generalize, which are really well understood concepts. We don't have to philosophize about what pattern matching really means.

[magicalhippo]

In the Physics of Language Models[1] they argue that you must augment your training data by changing sentences and such, in order for the model to be able to learn the knowledge. As I understand their argument, language models don't have a built-in way to detect what is important information and what is not, unlike us. Thus the training data must aid it by presenting important information in many different ways.

Doesn't seem unreasonable that the same holds in a gaming setting, that one should train on many variations of each level. Change the lengths of halls connecting rooms, change the appearance of each room, change power-up locations etc, and maybe even remove passages connecting rooms.

[1]: https://physics.allen-zhu.com/part-3-knowledge/part-3-1

[NBJack]

In other words, they learn the game, not how to play games.

[fsmv]

They memorize the answers not the process to arrive at answers

[EternalFury]

They learn the value of specific actions in specific contexts based on the rewards they received during their play time. Specific actions and specific contexts are not transferable for various reasons. John quoted that varying frame rates and variable latency between action and effect really confuse the models.

[nightpool]

Okay, so fuzz the frame rate and latency? That feels very easy to fix.

[IshKebab]

This has been disproven so many times... They clearly do both. You can trivially prove this yourself.

[0xWTF]

> You can trivially prove this yourself.

Given the long list of dead philosophers of mind, if you have a trivial proof, would you mind providing a link?

[beefnugs]

yeahhhh why isnt there a training structure where you play 5000 games, and the reward function is based on doing well in all of them?

I guess its a totaly different level of control: instead of immediately choosing a certain button to press, you need to set longer term goals. "press whatever sequence over this time i need to do to end up closer to this result"

There is some kind of nested multidimensional thing to train on here instead of immediate limited choices

[IshKebab]

Well yeah... If you only ever played one game in your life you would probably be pretty shit at other games too. This does not seem very revealing to me.

[trainerxr50]

I am decent at chess but barely know how the pieces in Go move.

Of course, this because I have spent a lot of time TRAINING to play chess and basically none training to play go.

I am good on guitar because I started training young but can't play the flute or piano to save my life.

Most complicated skills have basically no transfer or carry over other than knowing how to train on a new skill.

[drw85]

But the point here is, if i gave you a guitar with a string more or less. Or a different shaped guitar, you could play it.

If i give you a chess set with dwarf themed pieces and different colored squares, you could play immediately.

[e2021]

I don't think thats true. If you'd only ever played Doom, I think you could play, say, counterstrike or half-life and be pretty good at it, and i think Carmack is right that its pretty interesting that this doesn't seem to be the case for ai models

[antisthenes]

Where do you draw the line between pattern matching and reasoning about world models?

A lot of intelligence is just pattern matching and being quick about it.

[halfcat]

The line is: building an internal world model requires interfacing with the world, not a model of it, and subsequent failing (including death and survivorship over generations) and adaptation. Plus pattern matching.

Current AI only does one of those (pattern matching, not evolution), and the prospects of simulating evolution is kind of bleak, given I don’t think we can simulate a full living cell yet from scratch? Building a world model requires life (or something that has undergone a similar evolutionary survivorship path), not something that mimics life.

[BlueTemplar]

You don't need to simulate a full living cell to have evolution. In fact, isn't using evolving programs a decades-old technic ?

[halfcat]

Genetic programming models a natural process of evolution to do something useful, the same way machine learning models neurons to do something useful.

But producing something useful is a totally different thing from producing resilience in physical reality. That takes a world model, and I guess my suspicion is that an entity can’t build a world model without a long history of surviving in that world.

Put another way, you can never replicate what it’s like to burn your hand on the fire using only words. You could have a million people tell a child about what fire is like, the dangers of it, the power of it, the pain of it. But they will never develop an innate understanding of it that helps them navigate the real world.

Until they stick their hand in the fire. Then they know.

[ferguess_k]

I kinda think I'm more or less the same...OK maybe we have different definitions of "pattern matching".

[veqz]

It's Plato's cave:

We train the models on what are basically shadows, and they learn how to pattern match the shadows.

But the shadows are only depictions of the real world, and the LLMs never learn about that.

[ebonnafoux]

But the same is true for human, we get our information though our senses we do not have the __real__ word directly.

[actionfromafar]

We do much more than LLMs have. We have bodies and feelings.

[EternalFury]

100%

[Zanfa]

According to Carmack's recent talk [0], SOTA models that have been trained on game A don't perform better or train faster on game B. Even worse, training on game B negatively affects performance in game A when returning to it.

[0] https://www.youtube.com/watch?v=3pdlTMdo7pY

[rhdunn]

You can see a similar effect with LLM finetunes. If you finetune a base model (or other instruct/finetune model) for a new task (e.g. better maths or programming language comprehension) it performs worse at other tasks like creative writing.

To mitigate this you have to include the other categories in your finetune training dataset so it doesn't lose the existing knowledge. Otherwise, the backpropagation and training will favour weights that reflect the new data.

In the game example having the weights optimized for game A doesn't help with game B. It would be interesting to see if training for both game A and B help it understand concepts in both.

Similarly with programming languages it would be interesting to see if training it with multiple languages if it can extract concepts like if statements and while loops.

IIUC from the observations with multilingual LLMs you need to have the different things you are supporting in the training set together. Then the current approach is able to identify similar concepts/patterns. It's not really learning these concepts but is learning that certain words often go together or that a word in one language is similar to another.

It would be interesting to study multilingual LLMs for their understanding of those languages in the case where the two languages are similar (e.g. Scottish and Irish Gaelic; Dutch and Afrikaans; etc.), are in the same language family (French, Spanish, Portuguese), or are in different language families (Italian, Japanese, Swahili), etc.

[Zanfa]

> In the game example having the weights optimized for game A doesn't help with game B. It would be interesting to see if training for both game A and B help it understand concepts in both.

Supposedly it does both A and B worse. That's their problem statement essentially. Current SOTA models don't behave like humans would. If you took a human that's really good at A and B, chances are they're gonna pick up C much quicker than a random person off the street that hasn't even seen Atari before. With SOTA models, the random "person" does better at C than the A/B master.

[goatlover]

I've wondered about the claim that the models played those Atari/2D video games at superhuman levels, because I clearly recall some humans achieving superhuman levels before models were capable of it. Must have been superhuman compared to average human player, not someone who spent an inordinate amount of time mastering the game.

[raincole]

I'm not sure why you think so. AI outperforms humans in many games already. Basically all the games we care to put money to train a model.

AI has beat the best human players in Chess, Go, Mahjong, Texas hold'em, Dota, Starcraft, etc. It would be really, really surprising that some Atari game is the holy grail of human performance that AI cannot beat.

[tsimionescu]

I recall this not being true at all for Dota and Starcraft. I recall AlphaStar performed much better than the top non-pro players, but it couldn't consistently beat the pro players with the budget that Google was willing to spend, and I believe the same was true of Dota II (and there they were even playing a limited form of the game, with fewer heroes and without the hero choice part, I believe).

[wredcoll]

As I recall, the Starcraft ones heavily involved being able to exploit the computer's advantage in "twitch" speed over any human, it's just a slightly more complicated way of how any aim-bot enabled AI will always beat a human in an FPS, the game is designed to reward a certain amount of physical speed and accuracy.

In other words, the Starcraft AIs that win do so by microing every single unit in the entire game at the same time, which is pretty clever, but if you reduce them to interfacing with the game in the same way a human does, they start losing.

One of my pet peeves when we talk about the various chess engines is yes, given a board state they can output the next set of moves to beat any human, but can they teach someone else to play chess? I'm not trying to activate some kinda "gotcha" here, just getting at what does it actually mean to "know how to play chess". We'd expect any human that claimed to know how to play to be able to teach any other human pretty trivially.

[8n4vidtmkvmk]

I don't think the current chess models can train humans to play, but I imagine that's another thing that can be optimized for. Start with some existing chess training program, sprinkle in some AI, collect some data, figure out what methods increase ELO score the fastest.

[runeks]

> What John Carmack is exploring is pretty revealing. Train models to play 2D video games to a superhuman level, then ask them to play a level they have not seen before or another 2D video game they have not seen before.

Where can I read about these experiments?

[henryjcee]

some info in here https://www.slideshare.net/slideshow/john-carmack-s-slides-f...

[bob1029]

Generalization across tasks is clearly still elusive. The only reason we see such success with modern LLMs is because of the heroic amount of parameters used. When you are probing into a space of a billion samples, you will come back with something plausible every time.

The only thing I've seen approximating generalization has appeared in symbolic AI cases with genetic programming. It's arguably dumb luck of the mutation operator, but oftentimes a solution is found that does work for the general case - and it is possible to prove a general solution was found with a symbolic approach.

[hluska]

When I finished my degree, the idea that a software system could develop that level of expertise was relegated to science fiction. It is an unbelievable human accomplishment to get to that point and honestly, a bit of awe makes life more pleasant.

Less quality of life focused, I don’t believe that the models he uses for this research are capable of more. Is it really that revealing?

[bthornbury]

This generalization issue in RL in specific was detailed by OpenAI in 2018

https://arxiv.org/pdf/1804.03720

[moralestapia]

I wonder how much performance decreases if they just use slightly modified versions of the same game. Like a different color scheme, or a couple different sprites.

[fullshark]

Just sounds like an example of overfitting. This is all machine learning at its root.

[TimByte]

The gap between hype and actual generalization is still massive

[SquibblesRedux]

Indeed, it's nothing but function fitting.

[t55]

this is what deepmind did 10 years ago lol

[smokel]

No, they (and many others before them) are genuinely trying to improve on the original research.

The original paper "Playing Atari with Deep Reinforcement Learning" (2013) from Deepmind describes how agents can play Atari games, but these agents would have to be specifically trained on every individual game using millions of frames. To accomplish this, simulators were run in parallel, and much faster than in real-time.

Also, additional trickery was added to extract a reward signal from the games, and there is some minor cheating on supplying inputs.

What Carmack (and others before him) is interested in, is trying to learn in a real-life setting, similar to how humans learn.