[anon373839]

I disagree, but I also wasn’t referring to the exhaustion of training materials. I am referring to the fact that exponentially more compute is required to achieve linear gains in performance. At some point, it just won’t be feasible to do $50B training runs, you know?

[throw5959]

50B still seems reasonable compared to the revenue of the Big AI companies.

[mentalgear]

what revenues? If by big AI companies you mean llm service providers (OpenAI, ...), their revenues are far from high or profitable. https://www.cnbc.com/2024/09/27/openai-sees-5-billion-loss-t...

Maybe Nvidia, but they are a chip / hardware maker first. And even for them 50B training run with no exponential gains seems unreasonable.

Better to optimize the architecture / approach first, which also is what most companies are doing now before scaling out.

[throw5959]

It's not unusual to make infrastructure investments that will pay off in 30-50 years. I don't see why not an AI model - unless it's not true that we're at the end of scaling.