[noch]

> You seem to be assuming that the rapid progress in AI will suddenly stop.

> I think if you look at the history of compute, that is ridiculous. Making the models bigger or work more is making them smarter.

It's better to talk about actual numbers to characterise progress and measure scaling:

" By scaling I usually mean the specific empirical curve from the 2020 OAI paper. To stay on this curve requires large increases in training data of equivalent quality to what was used to derive the scaling relationships. "[^2]

"I predicted last summer: 70% chance we fall off the LLM scaling curve because of data limits, in the next step beyond GPT4.

[…]

I would say the most plausible reason is because in order to get, say, another 10x in training data, people have started to resort either to synthetic data, so training data that's actually made up by models, or to lower quality data."[^0]

“There were extraordinary returns over the last three or four years as the Scaling Laws were getting going,” Dr. Hassabis said. “But we are no longer getting the same progress.”[^1]

---

[^0]: https://x.com/hsu_steve/status/1868027803868045529

[^1]: https://x.com/hsu_steve/status/1869922066788692328

[^2]: https://x.com/hsu_steve/status/1869031399010832688

[ilaksh]

o1 proved that synthetic data and inference time is a new ramp. There will be more challenges and more innovations. There is a lot of room in hardware, software, model training and model architecture left.

[noch]

> There is a lot of room in hardware, software, model training and model architecture left.

Quantify this please? And make a firm prediction with approximate numbers/costs attached?

[ilaksh]

It's not realistic to make firm quantified predictions any more specific than what I have given.

We will likely see between 3 and 10000 times improvement in efficiency or IQ or speed of LLM reasoning in the next 5 years.

[noch]

> It's not realistic to make firm quantified predictions any more specific than what I have given.

Then do you actually know what you're talking about or are you handwaving? I'm not trying to be offensive but business plans can't be made based on a lack of predictions.

> We will likely see between 3 and 10000 times improvement in efficiency or IQ or speed of LLM reasoning in the next 5 years

That variance is too large to take you seriously, unfortunately. That's unfortunate because I was really hoping you had an actionable insight for this discussion. :(

If I, for instance, tell my wife I can improve our income by 3x or 1000x but I don't really know, there's no planning that can be done and I'll probably have to sleep on the couch until I figure out what the hell I'm doing.

[TeMPOraL]

> business plans can't be made based on a lack of predictions.

They can. It's called "taking a risk". Which is what startups are about, right?

It's hard to give a specific prediction here (I'm leaning towards 10x-1000x in the next 5 years), but there's also no good reason to believe progress will stop, because a) there's many low and mid-hanging fruits to pick, as outlined by GP, and b) because it never did so far, so why would it stop now specifically?

[jazzyjackson]

Why did we stop going to the moon and flying commercial supersonic?

Some things that are technologically possible are not economically viable. AI is a marvel but I'm not convinced it will actually plug into economic gains that justify the enormous investment in compute.

[noch]

> They can. It's called "taking a risk".

Spoken like a young man. I salute you. However, on your journey remember that risk of ruin is what you want to minimize relative to your estimated rewards. That is, not all risks can be afforded. I happen to have a limited budget, perhaps you don't and costs in terms of money and time don't matter for you.

Ruin can set you back years or decades or permanently and then you find yourself on a ycombinator thread hopelessly trying to find someone who can meaningfully quantity and forecast future medium term AI progress so that you can hire them to help your ongoing project. Alas all you get is the comments' section. :-)

> but there's also no good reason to believe progress will stop, because a) there's many low and mid-hanging fruits to pick, as outlined by GP, and b) because it never did so far, so why would it stop now specifically?

Specifically, due to lack of data. Please refer to the earlier comment[^0]: deep learning requires vast amounts of data. Current models have already been trained on the entire internet and corpus of published human knowledge. Models are now being trained on synthetic data and we're running out of that too. This data bottleneck has been widely reported and documented.

---

[^0]: https://news.ycombinator.com/item?id=42673410

[ben_w]

> If I, for instance, tell my wife I can improve our income by 3x or 1000x but I don't really know, there's no planning that can be done and I'll probably have to sleep on the couch until I figure out what the hell I'm doing.

For most people, even a mere 3x in the next 5 years is huge, it's 25% per year growth.

3x in 5 years is a reasonable low-ball for hardware improvements alone. Caveat: top-end silicon is now being treated as a strategic asset, so there may be wars over it, driving up prices and/or limiting progress, even on the 5-year horizon.

I'm unclear why your metaphor would have you sleeping on the sofa: If tonight you produce a business idea for which you can be 2σ-confident that it will give you an income 5 years from now in the range [3…1000]x, you can likely get a loan for a substantially bigger house tomorrow than you were able to get yesterday; in the UK that's a change slightly larger than going from the median average full-time salary to the standard member of parliament salary.

(The reason behind this, observed lowering of compute costs, has been used even decades ago to delay investment in compute until the compute was cheaper).

The arguments I've seen elsewhere for order-of-10,000x* cost improvements (which is a proxy for efficiency and speed if not IQ) is based on various different observations cost reductions** since ChatGPT came out — personally, I doubt that the high end of that would come to pass, my guess is those all represent low-hanging fruit that can't be picked twice, but even then I would still expect there to be some opportunity for further gains.

* The original statement had one more digit in it than yours, but this doesn't make much difference to the argument either way

** e.g. https://www.wing.vc/content/plummeting-cost-ai-intelligence