[SamFold]

I think it's somewhat of a stretch to say our "base model" had billions of years of evolution. Billions of years ago, mammals didn't exist and the only things around were more like plankton or algae and had nothing like a "base model" we could say we somehow inherited. The earliest ape-like creatures appeared around 10M years ago.

The first mammals appeared around 225M years ago so you could potentially argue that our "base model" first started evolving around then but I still think it's something of a stretch to compare this kind "training" to the ways we are training modern neural networks. The "base model" at this time was simply survival, eat, reproduce, survive, and enough circuitry to manage your base biological functions.

We are essentially running the entire volume of human knowledge through a neural network through billions of iterations and the model itself has 175 billion plus parameters. Humans nor any of our evolutionary ancestors never received this kind of "training", it's simply not comparable at all. Our mammalian ancestors were exposed to "basic" natural environments, they were not "pushed" into artifical situations to learn tool usage or language.

If we look at when apes first came about (10M years) ago and let's say since then the average ape or humanoid lived to 30-40 years, and estimate the average generation length for apes at 20 years (which is roughly accurate according to the latest research). This means that since the first recorded apes there have been about 500'000 generations of apes and humans. (12'000 generations for humans only).

So now if you compare how we are training our models, GPT-3 at 175B parameters and billions of iterations of training, GPT-4 we don't know. And again, extremely focused and specific training, feeding the entire human generated corpus of language, mathemetics, logic, etc etc into it, and we get something that does pretty well at human language.

Humans have a "base model" as you put it which really hasn't been trained for many generations and has been mostly exposed at random to external stimuli in an ad-hoc, unfocused way, and no single individual has ever been exposed to even a fraction of a fraction of a percent as much stimuli as a GPT model. So there is something different going on with our brain and neural networks and I think it can't really be compared at all: the mechanisms, numbers, and crucially, the results, do not match up in the slightest.

[cameronh90]

I was being slightly facetious in parts. The point I was driving at is that a human mind has had a very long time to do something perhaps akin to hyperparameter tuning, and we know that even imperceptibly minor changes in brain architecture can be the difference between struggling to put your socks on and being a genius that’ll be remembered through the ages. So those 500M years since the first neuron can’t just be written off.

Ultimately I agree with your final conclusion. You can’t really compare a LLM and evolved human directly. Even just a neuron in an ANN is nothing remotely comparable to a biological neuron. Of course it isn’t surprising that humans and LLMs are different given that they are built to do completely different things on fundamentally different hardware.

It just seems like many people are keen to write off the significance of GPT just because it’s not yet quite as good at everything compared to the world’s most marvellous example of engineering we all have in our skull. We didn’t even have transistors 75 years ago, but now we have a pretty believable facsimile (until you really interrogate it) of human intelligence that’s improving million times faster than evolution was ever capable of. But now the criticism is that it learns in a fundamentally different way to humans and doesn’t generalise fast enough. It’s true, but.. really?