[shrimpx]

Power is already concentrated in the sense that only the wealthy can train. We should make kickstarter-type models to fund open source AI training, because it'll likely continue to cost millions to tens of millions for cutting edge models.

With Llama2 we're at Meta's mercy, as it cost 20M to train. No guarantee Meta will continue to give us next-gen models. And even if it does, we're stuck with their training biases, at least to some extent. (I know you can fine-tune etc.)

[flockonus]

While i don't disagree, that's missing the forest for the trees - the point is having the freedom to share models and continue to open-source.

I'll argue that between stable diffusion and llama 2, there is nothing highly specific that prevents [very] large amount of people from adopting these models and specializing for them own needs.

The tragedy would be if those went away.

[shrimpx]

Yeah but llama2-level AI may be insignificant in power compared to future models, which may be inaccessible to the public. Even assuming the algorithms/code are open, people at large won't be able to create working models.

[flockonus]

Not sure at all, when you look at amount of opensource being done in regards to quantization. One of many examples: https://huggingface.co/TheBloke/Llama-2-7B-GGML

[autoexec]

> Power is already concentrated in the sense that only the wealthy can train.

That situation will change as technology evolves. We'll eventually reach a point where a normal desktop PC can train AI. The wealthy will always be able to do it faster, but the gap will shrink with time.

The trick is making sure that laws aren't put in place now that would restrict our ability to do that freely once the technology is there, and that massive data sets are compiled and preserved where the public has free access to them.

[CatWChainsaw]

Normal desktop PCs are going to be neutered to subscription-based terminals that you can take home with you rather than go to a central location to access.

[shrimpx]

That didn't happen with cloud though. I know it's not entirely apples to apples, but you still need a computer the size of a large building to serve Netflix, etc.

[lelandbatey]

Only if you're trying to serve netflix-quality stuff to hundreds of thousands of people. If you're trying to replicate "Netflix the product" (live video streaming with a slick interface) to a small set of individuals, you can do that with a personal computer (see Jellyfin, Plex).

Comparing Netflix (and most highly profitable computer businesses) to the world of producing AI models by training is not going to be fruitful. Netflix takes a lot of effort to operate but you can do on the small scale what Netflix does, quite directly. You can't replicate an AI model like ChatGPT-4 very easily unless you have all the data and huge compute that OpenAI does. Now, once the model has been produced, you can operate that model on the small scale with maybe less amazing results (see llama.cpp, etc) but producing the model is a scale problem like producing high quality steel. You can't escape the need for scale (without some serious technological developments first).

[autoexec]

Companies will always have an advantage with scale. It's not like you need a super computer though. You can have a single desktop media server in your home that does what netflix does without any problem. A single media server can even serve multiple homes.

Netflix cheats. They send non-supercomputer boxes out to ISPs to install locally. If I could convince every ISP to install a bunch of my media servers people could watch my shows from anywhere in the US too.

[shrimpx]

I guess we're speculating that training llama2 will drop by 1000x or something, so anyone can train their own llama2 from scratch for about $2k.

I don't think compute cost has dropped by 1000x since 20 years ago. Maybe by 10 to 50x. And if you add in the demand for higher quality, the cost has probably increased. Like encoding a video for streaming 20 years ago, at that standard, may have cost roughly the same as it does today, or more, when you factor in the increases in resolution and quality.

My prediction is that training the latest model will continue to cost millions to tens of millions for a long time, and these costs may even increase dramatically if significantly more powerful models require proportional increase in training compute.

Unless of course we have some insane algorithmic breakthrough where we find an AI algorithm that blows llama2 out of the water for a small fraction of the compute.

[PeterStuer]

What compute was available 20 years ago that would be the equivalent of an H100?

[shrimpx]

Fair point. I was thinking about compute in general. It could be that AI-specific compute cheapens more dramatically than CPU compute.