[BearOso]

I think LLM technology, not necessarily all of CNN, has plateaued. We've used up all the human discourse, so there's nothing to train it on.

It's like fossil fuels. They took billions of years to create and centuries to consume. We can't just create more.

Another problem is that the data sets are becoming contaminated, creating a reinforcement cycle that makes LLMs trained on more recent data worse.

My thoughts are that it won't get any better with this method of just brute-forcing data into a model like everyone's been doing. There needs to be some significant scientific innovations. But all anybody is doing is throwing money at copying the major players and applying some distinguishing flavor.

[theptip]

What data are you using to back up this belief?

Progress on benchmarks continues to improve (see GPT-o1).

The claim that there is nothing left to train on is objectively false. The big guys are building synthetic training sets, moving to multimodal, and are not worried about running out of data.

o1 shows that you can also throw more inference compute at problems to improve performance, so it gives another dimension to scale models on.

[KaiserPro]

> Progress on benchmarks continues to improve (see GPT-o1).

thats not evidence of a step change.

> The big guys are building synthetic training sets

Yes, that helps to pre-train models, but its not a replacement for real data.

> not worried about running out of data.

they totally are. The more data, the more expensive it is to train. Exponentially more expensive.

> o1 shows that you can also throw more inference compute

I suspect that its not actually just compute, its changes to training and model design.

[senko]

Actually, the sources we had (everything scraped from the internet) turns out to be pretty bad.

Imagine not going to school and instead learning everything from random blog posts or reddit comments. You could do it if you read a lot, but it's clearly suboptimal.

That's why OpenAI, and probably every other serious AI company, is investing huge amounts in generating (proprietary) datasets.

[chuckledog]

GitHub, especially filtered by starred repos, is a pretty high quality dataset.

[askafriend]

Any thoughts on synthetic data?

[rocho]

See "AI models collapse when trained on recursively generated data"

https://www.nature.com/articles/s41586-024-07566-y

[slashdave]

Dead end. You cannot create information out of nothing.

[NiloCK]

How did alpha zero succeed?

[slashdave]

By introducing new training data (train of thought) that was independently verified or explicitly constructed using external information.

[Filligree]

Which is why thought experiments are always useless.

[Lerc]

You're a creationist then?