[brcmthrowaway]

I could never get this about modern machine/deep learning or even the Transformers. Yes, it's not exactly rocket science, but when I see the data flow diagrams, it's not clear what is calculated in real time or multiple steps.

Is it really one big computation f(g(h(x)))?

[malwrar]

Yes.

Each token prediction is one big function call. Then you just recursively generate more tokens until run out of context or the model predicts a next token indicating end of sequence. Technically the model outputs a matrix where the last row is a probability distribution, but I’m counting sampling from it as part of the chain. Hundreds of billions of dollars has gone into just making the function fatter and gradually changing pieces here and there.

[brcmthrowaway]

I remember the concept of layers, as essentially defining the matrix math dimensions. And for a given model/framework, they were static. That always bugged me (not very dynamic).. is this still the case?

[malwrar]

The sizes for the matrices are decided up front (“hyperparameters”, more or less guesstimated) and the values that are learned (“parameters”, for completeness) and remain static after training. The layering just refers to the core “transformer block” being chained inside the model that contains most of the weights. Those themselves are pretty simple, usually just an MLP followed by an attention function, with variations on formulation & “multi-heading” in most of the new models. Behind the terms and math and inner architectural choices, it’s pretty much still the same GPT pattern of ‘“lie” = decode(sample(lm_head(T_1(T_2(…T_n(encode(“the cake is a”))))))’, with T_i(…) being the ith layer.

You know, I haven’t kept up with MoE and etc where there’s a bit of selection going on, so I should probably be more humble. I think new work has only added different “paths down the same hill” though (no recurrence, just select different matrices), but could be wrong there. I don’t think I’m wrong on my general intuition, just want to be epistemically honest!

[choilive]

What's your distinction between real time vs multiple steps? All computation is done in steps.

Is it all one big computation? Its turtles all the way down.

[leonvoss]

It's all vibes.