Hacker News new | ask | show | jobs
by gf000 90 days ago
Humans are "multi-modal". Sure we get plenty of non-textual information, but LLMs were trained on basically every human-written world ever. They definitely see many orders of magnitude more language than any human has ever seen. And yet humans get fluent based after 3+ years.
2 comments
int_19h 90 days ago
If you treat the human brain as a model, and account for the full complexity of neurons (one neuron != one parameter!) it has several orders of magnitude more parameters than any LLM we've made to date, so it shouldn't come as a surprise.

What is surprising is that our brain, as complex as it is, can train so fast on such a meager energy budget.

link
gf000 90 days ago
You are right, but at the same time the human brain does way more stuff (muscle coordination, smell, touch sensing) and all those others take up at least some budget.

So interesting question, but I'm not convinced it's only a scale issue. Like finished models don't really learn the same way as humans do - we actually change the parameters "at runtime", basically updating the model and learning is not only for the current context.

link
int_19h 83 days ago
It goes both ways though. All that extra stuff is also a part of our "training set" when growing up. And we have already seen that training models on vision etc improves their text outputs as well, even in tasks that aren't directly connected to visual things. That might account for a lot of our advantages.

But yes, of course it's not just a scale issue. Note though that a "finished model" can still be fine-tuned, and you can in fact allow it to fine-tune itself even. It's just that this is prohibitively expensive in practice (once again, the hardware is lagging behind the wetware here).

link
virgildotcodes 90 days ago
For sure, it seems like there's something there primed to pick up human language quickly, clearly evolutionarily driven.

Not necessarily so for the dynamics of magnetic fields, or nonhuman animal communications, or dark energy/matter.

We are bombarded nonstop by magnetic fields, nonhuman animal communications, and live in a universe which seems to be majority dominated by dark energy and matter, and yet understand little to none of it all.

link