[espadrine]

A short history:

1986: Geoffrey Hinton publishes the backpropagation algorithm as applied to neural networks, allowing more efficient training.

2011: Jeff Dean starts Google Brain.

2012: Ilya Sutskever and Geoffrey Hinton publish AlexNet, which demonstrates that using GPUs yields quicker training on deep networks, surpassing non-neural-network participants by a wide margin on an image categorization competition.

2013: Geoffrey Hinton sells his team to the highest bidder. Google Brain wins the bid.

2015: Ilya Sutskever founds OpenAI.

2017: Google Brain publishes the first Transformer, showing impressive performance on language translation.

2018: OpenAI publishes GPT, showing that next-token prediction can solve many language benchmarks at once using Transformers, hinting at foundation models. They later scale it and show increasing performance.

The reality is that the ideas for this could have been combined earlier than they did (and plausibly future ideas could have been found today), but research takes time, and researchers tend to focus on one approach and assume that another has already been explored and doesn’t scale to SOTA (as many did for neural networks). First mover advantage, when finding a workable solution, is strong, and benefited OpenAI.

[null_investor]

This is not accurate. OpenAI and other companies could do it not entirely because of transformers but because of the hardware that can compute faster.

We've had upgrades to hardware, mostly led by NVidia, that made it possible.

New LLMs don't even rely that much on that aforementioned older architecture, right now it's mostly about compute and the quality of data.

I remember seeing some graphs that shows that the whole "learning" phenomena that we see with neural nets is mostly about compute and quality of data, the model and optimizations just being the cherry on the cake.

[espadrine]

> New LLMs don't even rely that much on that aforementioned older architecture

Don’t they all indicate being based on the transformer architecture?

> not entirely because of transformers but because of the hardware

Kaplan et al. 2020[0] (figure 7, §3.2.1) shows that LSTMs, the leading language architecture prior to transformers, scaled worse because they plateau’ed quickly with larger context.

[0]: https://arxiv.org/abs/2001.08361

[nimithryn]

Also, this sort of thing couldn't be done in the 80s or 90s, because it was much harder to compile that much data.

[camjw]

I know this is just a short history but I think it is inaccurate to say "2015: Ilya Sutskever founds OpenAI." I get that we all want to know what he saw etc and he's clearly one of the smartest people in the world but he didn't found OpenAI by himself. Nor was it his idea to?

[trashtester]

Ilya may not be the only founder. Sam was coordinating it, Elon provided vital capital (and also access to Ilya).

But out of the co-founders, especially if we believe Elon's and Hinton's description of him, he may have been the one that mattered most for their scientific achievements.

[espadrine]

Short histories remove a lot of information, but it would be impractical to make it book-sized. There were numerous founders, and as another commenter mentioned, Elon Musk recruited Ilya, which soured his relationship with Larry Page.

Honestly, those are not the missing parts that most matter IMO. The evolution of the concept of attention across many academic papers which fed to the Transformer is the big missing element in this timeline.

[jll29]

> but it would be impractical to make it book-sized

Not really:

History: https://arxiv.org/abs/2212.11279 (75 pp.)

Survey: https://arxiv.org/abs/1404.7828 (88 pp.)

Conveniently skim-read over the course of the four weekends on one month.

[blackeyeblitzar]

I thought Elon Musk is who personally recruited Ilya to join OpenAI, which he funded early on, alongside others?

[flkenosad]

What a time to be alive!