[carlmr]

While this is true, the complexity perspective misses something more fundamental.

1) Our brains, and moreso those of animals, come with a really good pretraining at birth. This is collective genetic knowledge of millions of generations distilled into your brain.

2) Our brains have a lot of sensors and actuators to interact with the world. We only learn by reading as adults when our brains can already do the synesthesia of translating words into thought. But even as adults, most of us learn better if we do something, write something, engage in dialog, instead of passively listening, reading, or watching.

Passive data can never replicate the rich environment our brains grow up in.

[beisner]

While true, there’s a relatively small upper bound on how many bits of information are in this pre-training. Specifically, in the form of how much information is contained in DNA, which is only a couple gigabytes.

[lossolo]

Stable diffusion model is around 4 gigabytes, inside that 4 gigabytes you have understanding of the whole english language model and mapping to billions of objects, people, concepts etc capable of generating from just a single sentence almost any picture in any style you imagine. Seems like a few gigabytes can hold a lot of information.