[monodeldiablo]

It's not really even a question. It's an obvious boondoggle. The forecasted net new energy requirements for the AI buildout over the next couple of years are roughly equivalent to all of Western Europe's power demand today.

That's absurd. It's a physical impossibility to bring that much power online that quickly. And the cost to get even close would make AI more expensive than just hiring knowledge workers to do the same tasks.

And it's all predicated on a tower of wobbly or broken assumptions -- chief among them that increasing the size of these models yields better performance.

We're going to look back on this era and wonder why anybody took any of the outrageous claims of tech CEOs seriously.

[aoeusnth1]

> Wobbly assumption that increasing the size of these models yields better performance.

I'm assuming you disagree that larger models are better? Can you expand on what indicates that AI will hit a wall in scaling given the evidence of the last 9 years of scaling transformers (or other models)? Where on the plot does the line go from exponential to flat?

[monodeldiablo]

Leaks from within OpenAI have made it pretty clear that they've been struggling to achieve significant improvements lately by simply scaling up parameter size. Experts like LeCunn have also been vocal that blindly scaling up is a dead end.

(Incidentally, the line of skill improvement isn't "exponential". It's been incremental in improvements per generation, but generations have been coming thick and fast of late, and have grown in parameter count exponentially since 2017.)

Speaking more broadly, LLMs don't have to "hit a wall" in scaling to become uneconomical. If incremental improvement continues to come at exponential cost, however, then the fundamental value argument falls apart.

Setting all that aside, even presuming that model performance scales linearly with dimensionality, there are just fundamental limits to the size of the training corpuses. Quality training data is not unbounded and infinite. Given the same size corpus of training data, there's a hard theoretical limit to how much meaning and inference a model can wring out of it.

And then there are other issues with the whole business model. For one thing, it's predicated on continual full scale retraining to achieve even modest gains in skill and relevancy. Topical and targeted learning requires a full retraining. Etc cetera.

I think that the next generation of AI will lean more heavily on RL to be useful beyond a few months. I also think that the energy requirements of a particular technology are a good proxy to whether it's got a realistic future.

[emp17344]

Why do you believe progress is currently exponential? There’s one dubious chart showing “exponential growth” in a single narrow domain, and otherwise zero evidence to suggest exponential improvement.

[danaris]

The evidence is the last 9 years of scaling.

The curve flattened out years ago. The exponential was going from GPT-2 to GPT-4 (or thereabouts). After that, it was painfully obvious to anyone observing without a vested interest in believing otherwise that the progress had slowed.

Now, it's not just that progress has slowed: it's that the exponential has reversed. In order to get marginal gains, they have to throw exponentially more hardware at the training.

[functional_dev]

even if traning is hitting a wall I think they are shifting more to reasoning phase to get better results... and that is inference compute scaling

[righthand]

In my experience the models havent gotten any better, just the hype.

[heavyset_go]

And companies know this hence the heavy astroturfing, if their new product has minimal improvements they'll just gaslight you into thinking otherwise

[1vuio0pswjnm7]

"We're going to look back on this era and wonder why anybody took many of the outrageous claims of tech CEOs seriously."

It's the money. Without it, "no one" would take these ideas seriously. We know this because before the money "no one" took such ideas seriously

[jen729w]

> It's a physical impossibility to bring that much power online that quickly.

China begs to differ.

[monodeldiablo]

I played a role in China's shift to renewables. It's been decades in the making.

[tim333]

It quite likely won't become a $9T bust though because the investment so far is more like $0.8T. If things slump shortly it might be more like a $0.5T bust if you assume the things are worth 50% of what is being paid - there are a lot of paying users.

[deckar01]

They could get lucky, make a break through in robotics, and vertically integrate power generation into their business model with minimal human labor.