[threeseed]

> 10x better, 10x faster, and 10x cheaper

Which is the elephant in the room.

There is no roadmap for any of these to happen and a strong possibility that we will start to see diminishing returns with the current LLM implementation and available datasets. At which point all of the hype and money will come out of the industry. Which in turn will cause a lull in research until the next big breakthrough and the cycle repeats.

[Sysreq2]

While we have started seeing diminishing returns on rote data ingestion, especially with synthetic data leading to collapse, there is plenty of other work being done to suggest that the field will continue to thrive. Moore’s law isn’t going anywhere for at least a decade - so as we get more computing power, faster memory interconnects, and purpose built processors, there is no reason to suspect AI is going to stagnate. Right now the bottleneck is arguably more algorithmic than compute bound anyways. No one will ever need more than 640kb of RAM, right?

[thwarted]

I feel like the GP and this response are a common exchange right before the next AI Winter hits.

[__loam]

https://cap.csail.mit.edu/death-moores-law-what-it-means-and...

[threeseed]

a) It's been widely acknowledged that we are approaching a limit on useful datasets.

b) Synthetic data sets have been shown to not be a substitute.

c) I have no idea why you are linking Moore's Law with AI. Especially when it has never applied to GPUs and we are in a situation where we have a single vendor not subject to normal competition.

[nl]

Synthetic data absolutely does work well for code.

While Moore's Law probably doesn't strictly apply to GPUs, it's not far off. See [1] where they find "We find that FLOP/s per dollar for ML GPUs double every 2.07 years (95% CI: 1.54 to 3.13 years) compared to 2.46 years for all GPUs." (Moore's law predicts doubling every 2 years)

https://epochai.org/blog/trends-in-gpu-price-performance#tre...

[dartos]

It’d be really nice to see research in this area from somewhere without a financial interest in hyping AI.

That incentive doesn’t invalidate research, but AI results are so easy to nudge in any direction that it’s hard to ignore.

[Agentus]

I wonder when people mention Moores law do they use that vernacular literally or figuratively. IE literal as having to do with shrinking of the transistors, figuratively with any and all efforts to increase overall computational speed up.

[dartos]

In this context it’s the latter, but practically speaking they’re the same thing.

[FeepingCreature]

b is made up. They have absolutely not been shown to not be a substitute. It's just a big flood of bad research which people treat as summing up to a good argument.

[viraptor]

Maybe not 10x yet, but deepcoder has done some impressive things recently. Instead of a generic LLM, they have a relatively smaller one which is coding specific and gpt4-class in quality. This makes it cheaper. In addition, they can do caching which ~10x reduces the cost of follow-up request. And there are still improvements around Star, which reduces the need for learning datasets (models can self-reflect and improve without additional data)

So while we're not 10x-ing everything, it's not like there's no significant improvements in many places.

[viraptor]

I meant deepseek coder. Can't edit anymore.

[tmikaeld]

Unfortunately the smaller model is not anywhere near GPT4 in quality and no one seems to want to host the bigger model (it was even removed from fireworks ai this week). And no one in their right mind want to send their code to deepmind chinese API hosting.

[viraptor]

I'm perfectly fine sending my open source code to them. I'm also happy to send 95% of my private repos. Let's be honest, it's just boilerplate code not doing anything fancy, just routing/validating data for the remaining 5%. Nobody cares about that and it's exactly why I want AI to handle it. But I wouldn't send that remaining 5% to OpenAI either.

[Agentus]

Much of nvidias marketing material covers this if you want to believe it. They at minimal claim that there will be a million fold increase in compute available specifically to ML over the next decade.

[krageon]

You don't know where it will go, just as people didn't know the development of LLMs at all would happen. There are no real oracles to this level of detail (more vaguely in broad lines and over decades some Sci-Fi authors do a reasonable job, and they get a lot wrong).

There have been a lot of people making these sorts of claims for years, and they nearly never end up accurately predicting what will actually happen. That's what makes observing what happens exciting.

[jejeyyy77]

Actually the improvement graphs are still scaling exponentially with training/compute being the bottleneck. So there isn't yet any evidence of diminishing returns.

source: https://youtu.be/zjkBMFhNj_g?feature=shared&t=1545

[Agentus]

I just viewed an Andrew NG video (he is the guy i tended to learn the latest best prompting, agentic, visual agentic practices from) that hardware companies as well as software are working on making these manifest especially at inference stage.

[GregOz]

Can you include link to Andrew NG's video please.

[Agentus]

I think this was the relevant video not 100% sure. https://www.youtube.com/watch?v=8lH1mUcxODw&t=2013s