[aspenmartin]

I really wish more people skeptical of AI capabilities would read about scaling laws -- Lilian is always so marvelous at giving a deep overview of the technical side but the whole point of this is: there are scaling laws, and they hold and continue to hold. This is such a huge basis for the predictions about AI capabilities for the past like 5 years.

[an0malous]

Why should the skeptics be reading it? The scaling laws show diminishing returns on more training data and larger models.

From the Kaplan scaling laws paper:

> We have observed consistent scalings of language model log-likelihood loss with non-embedding parameter count N, dataset size D, and optimized training computation Cmin, as encapsulated in Equations (1.5) and (1.6). Conversely, we find very weak dependence on many architectural and optimization hyperparameters. Since scalings with N,D,Cmin are power-laws, there are diminishing returns with increasing scale.

So the skeptics are right to be skeptical of LLMs being all you need for continued advancement in this space. It seems like the believers are the ones who need to learn about the scaling laws.

[aspenmartin]

> We have observed consistent scalings of language model log-likelihood loss with non-embedding parameter count N, dataset size D, and optimized training computation Cmin, as encapsulated in Equations (1.5) and (1.6). Conversely, we find very weak dependence on many architectural and optimization hyperparameters. Since scalings with N,D,Cmin are power-laws, there are diminishing returns with increasing scale.

This is why, because this completely misses the point. Every power law has diminishing marginal returns, that is by construction. The point is that they diminish predictably and smoothly across seven orders of magnitude of compute. There is no plateau anywhere in the observed range, and in fact larger models more sample efficient than expected.

You can't just cherry pick one sentence containing the phrase "diminishing returns" and ignore the actual paper. The whole reason the labs spent the GDP of a small country on GPUs is that the curve is boring and reliable. No one just was like "oops we missed that line in the Kaplan paper, shit".

Lots of arguments as to why things may slow down or hit fundamental limits requiring a paradigm shift. But at several points in the past 5 years many major players have assumed this to be the case and have been continually proven wrong. Not to mention the plethora of socioeconomic horrors coming out of this Pandora's box, which I believe to be a large colocated B300 compute cluster somewhere in the midwest.

A point that would have been consistent with Kaplan was them finding alpha ~ 0.05 on compute means halving the reducible loss is a million times more expensive. But this was invalidated by Chinchilla who found it to be ~0.36 (100x more compute, not 1million). And loss has a non-linear relationship with capabilities, but roughly halving pretraining loss is ~doubling the METR task horizon. We also haven't factored in something incredibly important which is: algorithmic / architectural / training pipeline gains. These are substantial too!

[FromTheFirstIn]

And sitting right next to the data and compute factors in every cross entropy loss equation is the entropy of the language, which is just a fixed constant. There’s such a hard cap on cross entropy loss training and I never hear it come up!

[aspenmartin]

Right but that is context dependent; it drops with context length, depends on tokenizer, etc. It doesn't end up being super relevant, despite the fact that if you look at the loss for real models it's relatively large in absolute terms. But that doesn't really matter -- all of the interesting stuff happens once you start getting closer and closer to it. You've gotten past all of the easy tokens that dominate the entropy and now you get to the really challenging ones that we care about (like e.g. very difficult reasoning about a next step).

[FromTheFirstIn]

My understanding is that the true entropy floor of a language is intractable- regardless of context length there will be “unpredictable” tokens where cross entropy loss is bound to happen. Even with infinite parameters and data you’ll still have a chance at failing to predict the next token correctly a decent chunk of the time.

Also, linear gains in context length scale quadratically with compute because of attention, so depending on context growth means taking a bath on GPUs for as long as you can, right?

[graboy]

Yeah I mean, if you and I were to play the word-guessing game where you needed to guess what next word I'm thinking of, there's always uncertainty in your guess because it's a game of partial information - you can't fully observe my inner state. But that doesn't mean you couldn't evolve a strategy that spends a really long time thinking and analyzing to get asymptotically close to the best guess. There's no limit on that intelligence.

[FromTheFirstIn]

Isn’t the limit exactly what you’re describing? There’s always uncertainty, and your asymptote can approach its limit but it does have a limit. That’s the limit to the intelligence. And this is just for cross entropy loss- even if you could get loss to 0, I’m still not convinced at all that an enormous semantic map and its convoluted geometries amounts to intelligence.

[aspenmartin]

If you get to E you have generated a Bayes-optimal model of the conditional distribution (as in, next token conditional on context). This is something I thought too, but even if you're a fraction of a nat above the floor, you could have enormous headroom in performance left because there are still rare tokens amongst the irreducible noise that require so much capability to predict. It's not to suggest there truly is no cap on capability, but just that this constant isn't really saying what that is.

[krackers]

Isn't that one of the reasons why KL-divergence is used, at least in DPO/RL for LLM? Otherwise the model can effectively cheat and mode collapse. For pre-training against a 1-hot label the KL-divergence should be equivalent to cross-entropy anyway.

[317070]

Right, and what happens at that limit is most exciting! A model that has a cross entropy at that limit for a data stream of text, produces a stream of text that is both theoretically and practically indistinguishable from the original stream.

And so if the datastream has been produced by something intelligent, the resulting model is indistinguishable from that intelligence. That is the whole compression idea behind artificial intelligence.

The limit is not a bug, it's a feature!

[FromTheFirstIn]

It’s a bit like saying copy and paste must be intelligent because its output is indistinguishable from intelligent output. Reproduction of intelligence has never been equivalent to intelligence- that’s why we look down on copying, plagiarism or derivative work.

[317070]

Well if it was the data that was copied, nobody would be using LLMs. The data-generating process is the thing that would be copied.

That's why we do appreciate the nth artist making classical or techno music.