[autoexec]

Companies will always have an advantage with scale. It's not like you need a super computer though. You can have a single desktop media server in your home that does what netflix does without any problem. A single media server can even serve multiple homes.

Netflix cheats. They send non-supercomputer boxes out to ISPs to install locally. If I could convince every ISP to install a bunch of my media servers people could watch my shows from anywhere in the US too.

[shrimpx]

I guess we're speculating that training llama2 will drop by 1000x or something, so anyone can train their own llama2 from scratch for about $2k.

I don't think compute cost has dropped by 1000x since 20 years ago. Maybe by 10 to 50x. And if you add in the demand for higher quality, the cost has probably increased. Like encoding a video for streaming 20 years ago, at that standard, may have cost roughly the same as it does today, or more, when you factor in the increases in resolution and quality.

My prediction is that training the latest model will continue to cost millions to tens of millions for a long time, and these costs may even increase dramatically if significantly more powerful models require proportional increase in training compute.

Unless of course we have some insane algorithmic breakthrough where we find an AI algorithm that blows llama2 out of the water for a small fraction of the compute.

[PeterStuer]

What compute was available 20 years ago that would be the equivalent of an H100?

[shrimpx]

Fair point. I was thinking about compute in general. It could be that AI-specific compute cheapens more dramatically than CPU compute.