[antirez]

There is a different way to look at this: that is, actually the Transformer is a minimal complication of what the based model is: in theory the neural network could be just a huge FFN, which is anyway the part of the Transformer that does the heavy lifting. But this would be impossibile to train both numerically and computationally, so the Transformer encodes enough priors for it to work: the causal attention, and the math tricks like the residuals and so forth. But the bottom line of all this is that the Transformer works because of the incredible semantical power of simple/huge FFNs.

[dist-epoch]

Isn't that over-simplifying it a bit too much?

You can go another step - a FFN can be simulated on a Turing machine, thus it just exemplifies the incredible semantical power of the Turing machine model of computation. (in fact you don't even need a Turing machine, since there is no looping in one forward pass).

In theory you can run a huge FFN on the tiniest Turing machine, in practice it's much better to run a Transformer on the latest NVIDIA hardware. Or as they say "quantity (performance) has a quality all its own"

[musebox35]

I was about to post your last point / quote. Going multigpu is relatively not so though but once you go multi-node you have distributed storage/io/compute system which is highly non trivial. Add that the long training times now you have robustness/fault-tolerantness concerns with hardware failures and restarts. Today’s training systems are engineering marvels.

[zbendefy]

Good point!

There is also the case for Markov chains being theoretically able to do these if tuned well. Or even SAT problem.

[CGMthrowaway]

"LLM is just fancy autocomplete"

[galaxyLogic]

LLM is an Oracle

[Chu4eeno]

Well, it seems obvious (in hindsight) that attention is vital to actually do anything useful with all those matrices getting smashed together in increasingly unhinged ways in more and more layers.

Which is incidentally more or less the only thing I remember about Global Workspace Theory (attention facilitating consciousness in a way iirc).