[int_19h]

If you treat the human brain as a model, and account for the full complexity of neurons (one neuron != one parameter!) it has several orders of magnitude more parameters than any LLM we've made to date, so it shouldn't come as a surprise.

What is surprising is that our brain, as complex as it is, can train so fast on such a meager energy budget.

[gf000]

You are right, but at the same time the human brain does way more stuff (muscle coordination, smell, touch sensing) and all those others take up at least some budget.

So interesting question, but I'm not convinced it's only a scale issue. Like finished models don't really learn the same way as humans do - we actually change the parameters "at runtime", basically updating the model and learning is not only for the current context.

[int_19h]

It goes both ways though. All that extra stuff is also a part of our "training set" when growing up. And we have already seen that training models on vision etc improves their text outputs as well, even in tasks that aren't directly connected to visual things. That might account for a lot of our advantages.

But yes, of course it's not just a scale issue. Note though that a "finished model" can still be fine-tuned, and you can in fact allow it to fine-tune itself even. It's just that this is prohibitively expensive in practice (once again, the hardware is lagging behind the wetware here).