[remuskaos]

I think it's supposed to quite the opposite, at least it was like that for me. When gpt3 first launched, it seemed almost like magic for me, and I have some limited ml background too. Much later I saw the 3blue1brown Video about transformers, and I was almost disappointed to see the math itself is rather simple.

My main take away was that even simple basics can produce astonishing results, I this case if they're just scaled large enough. That incredibly complex and useful emergent behavior can result from what seems like conways-game-of-life like principles.

[goatlover]

It's the large training data which contains the knowledge for that complex and useful emergent behavior. It's like if you could import vast information about the world into Conway's game of life to enable increasingly complex levels of emergence.

[elcritch]

Oh idea of "vast patterns" in Conway's game of life seems like a great metaphor for LLM transformers. Explaining hidden markov chains with some extras or whatnot to a layman doesn't give a good mental image but is what I guess it's like.

Probability chains alone don't seem to give a good mental visualization of how such a system comes to certain "decisions" or "thought patterns". But watching the Game of Life you can see fascinating patterns which emerge and lead to interesting patterns. That's easy to extrapolate.

Maybe in the future NNs will be understable sorta like Game of Life, "oh that NN section is running pattern 27 on XY input data. That's gonna be an unstable behavioral element combined with pattern 38c over here." Not sure if that's a fascinating or dreadfully boring prospect though.