[Lomlioto]

Compression is the trick. Its even philosophed about if compression = intelligence.

The LLM has to compress everyy question/prompt into its system. It does so by creating rules and ways of processing data (this can lead to AGI, world models or an architecture of sub architectures like an LLM + something else). So if it should respond in a way that only reasoning people can achieve, it might be able to learn a representation of what we call reasoning.

It read enough text in itself to even know about the concept of reasoning and how you would do that.

Even if this is only stochastic, it shouldn't be so devalued as your comment comes across.

Who says that we are doing anything more magic?

[21asdffdsa12]

So, the best of compression is a procedural system, capable to generate from minimal input a whole world- basically just using abstract rules and metrics for reasonableness? So- AGI is in the principles behind universe generators like NoMansSky? Is there a ultimate game- that generates this from physics up?

[Lomlioto]

Yes + the capability to use this 'world' to simulate outcomes. Thats exactly what LeCun and others are trying to do with world models.

NoMansSky generator misses the complexity of encoding more than just geometry and basic quests but thats one reason why google and others show you machine learning based 'game engines' which allow you to enter these worlds. They are not doing this to replace real game engines, the do this for world research.

The ultimate game is us. Our physics constants :D

[warumdarum]

Compression, but also generating , using experience and previous abstractions and evaluating new state from entropy by comparing it to previous discoveries