[brucethemoose2]

Right now training is insanely expensive because Nvidia, but I don't think that's sustainable given the demand. Eventually, training hardware may be priced like commodity CPU instances, and won't require a bajillion infiniband-linked nodes for respectable throughput.

But for now... I think you have a point. We would have seen more than Falcon, MPT, Llama, and the open Llama reproductions by now if open source foundational model training was viable.

[adventured]

The training will always be extremely expensive and beyond the grasp of commodity computing at a low'ish cost.

The reason for that, is that the push will always be one direction in terms of size and improvement. That isn't going to stop. It will continue to push the edge of resources.

It's not Nvidia that's holding back the premise. AMD and Intel also can't do anything for you beyond what Nvidia can.

One might proclaim: yeah but in five years you'll be able to train CodeLlama 34B on a modest commodity desktop. Nobody will want to do that at that point, they'll want access to CodeLlama 204B.

The money will be in hosting these as a service business, and building further ecosystems around that. Not much different than the way a lot of major open source oriented companies have made their money, despite their core product being largely free to use.

[cbozeman]

I think you're correct in the "right now", but I think things fall off dramatically in the next 5-10 years. NVIDIA's progress from 2018 to 2023 was insanely impressive. AMD's was too in fact, at least technologically. Intel is third, but at least still in the game.

Where your premise here falls apart, is the age-old adage, "Don't let perfect be the enemy of the good."

We probably don't need CodeLlama 204B to be highly effective in our jobs as developers. Just like physicians probably don't need DocLlama 566B to be more effective in their jobs. We only need good enough. Good enough to be useful and insightful; not the All-Seeing Oracle that has insight beyond mortal man.

I think that revolution is probably coming sooner than we think. I'd argue that in 5-10 years, we'll have hardware powerful enough that a motivated and moderately well-heeled consumer can train their own LLM with domain-specific knowledge and have a product as useful as Llama8 1.3T or ChatGPT-14. There'll always be a market - I think anyway - for incredibly powerful general-purpose LLMs that are subscription-based (whatever that looks like in terms of cost / month), but locally trained, locally run LLMs designed to do one thing and do it well? I think that's the real money and the real future.

[a_wild_dandan]

Well said. Companies will want:

* Pricing stability -- your business costs cannot be at the whim/survival of some particular company. Competition (including local options) is the solution.

* Model stability –– model output must be reproducible. If it's shifting unreliably, that's a showstopper.

* Alignment –– The model I use must be aligned to my business, not to Some AI Business. I don't want a model that was kneecapped in some secret, arbitrary way that a group of largely white, male, West Coast techbros decided was crucial for my business.

* Privacy –– Sensitive company data NEVER leaves the building. It sure as hell doesn't get expedited to a company with strong incentives to hoover up and use that data.

There will soon be 3 big competitors racing each other in the GPU space. Specialized AI hardware will grow in volume and power. Algorithms will be optimized. Perhaps new algorithms will appear for distributed training of open source models by millions of consumers cards (reminiscent of the Folding@home project). Hardware 5 years from now will blow away current tech.

Intelligence is a widely applicable asset with strong incentives to pop up everywhere. It won't sit gated behind a handful of companies. That's true regardless of substrate -- proteins or silicon.

[cbozeman]

I agree with your assessment on GPU sustainability.

Simply put, AMD is going to have to get their shit together. NVIDIA was embraced by the AI community so long ago because CUDA isn't that tough to use. I haven't looked at AMD's solutions at all - in fact I don't even know what they call their version of CUDA - but I know one thing for certain is that the world cannot depend on a single company to supply the underlying technology for this technological revolution.

That's an easy way to a dystopian hellscape.

And the only reason we haven't seen open source foundational model training is, because as you said, the only significant player in the arena right now is NVIDIA.

Here's hoping Lisa Su is shifting priorities - and quickly.

[cryptobruh]

You still need the data.

And eventually might mean 10 years.

4k took ages until it just worked. I even switched to the M1 for this

[progbits]

4k as in screen resolution? That was a thing for many many years before M1.

[cryptobruh]

Of course but I still had laggy ui and issues when sharing my screen etc.

For example the gcp (Google cloud ui) became unusable while sharing in Ms teams.

Also good 4k displays/fast ones still had a fan build in.