|
|
|
|
|
|
by mikepalmer
1203 days ago
|
|
|
Regarding Chomsky's characterization of LLMs "gorging on hundreds of terabytes of data" compared to the "miniscule data" required for a child to learn a language: The brain is evolved, so the the "gorging" already happened in animals for 100s of millions of years. The brain has a lot of evolved sequence processing, visual processing, language (the authors are linguists and they admit this though it undercuts their point). Only fine tuning of this pretrained model is needed for a child to grow up speaking, say, English vs. French. This requires only a relatively miniscule amount of data. Moreover, it doesn't matter that LLMs work differently from the human brain. Per Larry Wall, TIMTOWTDI ("There is more than one way to do it"). |
|
|
Most animals don't acquire language, and the non-humans who are the best candidates for it all appear to have developed it independently; specifically in humans the potential is at most around 10mil years ago since we don't find it in other closely-related primates. Actual language is ~3 orders of magnitude younger, and during most of the time since then humans didn't have such high-density ways to ingest it. As I mentioned elsewhere in the comments, GPT-3 consists of around a million modern-human-years of linguistic intake. It seems to have the strong lead here.
> it doesn't matter that LLMs work differently from the human brain.
It does if your goal is to learn how human language acquisition works, rather than grind through another trillion in VC cash via your mechanical parrots.