[Lerc]

The shovels and labour used to make those things where not depreciated.

The GPUs are the shovels, not the project. AI at any capability will retain that capbibilty forever. It only gets reduced in value by superior developments. Which are built upon technologies that the previous generation developed.

[kennywinker]

Calling the GPUs the shovels is bonkers because a) shovels are cheap, GPUs are not. And b) when you build a bridge the bridge doesn’t need shovels to be passable. Without GPUs, the datacenter is useless, the model is useless, etc.

If anything, the GPUs are the steel that the bridge is made of. Each beam can be replaced, but if too many fail the bridge is impassible. A bridge with a 6 year lifespan for each beam is insane.

[throwup238]

You’re taking the metaphor way too literally. The people who made the most profit weren’t literally selling shovels, they were the ones providing logistics and support services to the gold miners, like hauling tons of equipment over tens of miles of mountain or providing the sales channel for the gold. They siphoned off most of the profit from the ventures that depended on them (like LLMs depend on GPUs) because the miners had no other choice, to the point where even the most productive mines often weren’t profitable at all.

A less literal example is the conquistadors: their shovels were ships, horses, gunpowder, and steel. You can look at Spanish records from the Council of the Indies archive and any time treasures were discovered, the price of each skyrocketed to the point where only the wealthiest hidalgos and their patrons could afford to go on such adventures. I.e. the cost of a ship capable of a cross Atlantic voyage going from 100k pieces of eight to over a million in the span of only a few years (predating the treasure fleet inflation!)

Gold rushes create demand shocks, and anyone who is a supplier to that demand makes bank, regardless of whether its GPUs or “shovels”.

[TeMPOraL]

> You can look at Spanish records from the Council of the Indies archive and any time treasures were discovered, the price of each skyrocketed to the point where only the wealthiest hidalgos and their patrons could afford to go on such adventures.

Today this is real estate. And it's something people keep forgetting when arguing that ${whatever breakthrough or just more competition} will make ${some good or service} cheaper for consumers: prices of other things elsewhere will raise to compensate and consume any average surplus. Money left on the table doesn't stay there for long.

[zozbot234]

GPUs don't really have six year lifespans, though. The hardware itself lasts far longer than that, even hardware that's been used for cryptomining in terrible makeshift setups is absolutely fine for reuse.

[malfist]

Each of these GPUs pull up to a kilowatt of power. The average commercial power cost is 13.4 ¢/kWh. That means running a single H100 full tilt 24/7 is a power operationing cost of $1,100 per card per year.

In three years the current generation of GPUs will be 50% or more faster. In six years your talking more than 100% faster. For the same energy costs.

If you're running a GPU data center on six year old GPUs, your cost to operate per sellable unit of work is double the cost of a competitor.

[Ekaros]

One thing I am not entirely sure if there will be huge efficiency gains. Just looking at TDP that is the power consumption of say 3090 and 5090 and the increase is substantial then compare it to performance and the performance lift stops looking that great...

[xyhopguy]

3x increase in compute for a 1.5x increase in tdp is pretty good considering the underlying process had barely changed. In anycase, consumer GPUs aren't a good metric as they operate with different economic constraints.

H100 to GB200 saw a 50x increase in efficiency, for example.

[Ekaros]

https://www.nvidia.com/en-us/data-center/gb200-nvl72/

Nvidia only advertises 25x efficiency. And that is their word...

[brookst]

Sure. But if that fully depreciates, $1100/year GPU produces $20k of economic benefit, would you decommission it as long as there is demand?

[malfist]

If my data center sells a pflop at $5 because of our electricity use and the data center a state over with newer GPUs sells it at $2.50/pflop, it doesn't matter how much economic benefit it generates, my customers are all going to the data center a state over.

[voakbasda]

I want to see math on how a single GPU will pull down that much revenue, because that seems like a dubious outcome.

[brookst]

Fair, I was hand waving to make a point. “If it generates more than $1100 + (resale price * WACC) + opportunity cost from physical space/etc” would have been more accurate.

But the point is — you don’t decommission profit generators just because a competitor has a lower cost structure. You run things until it is more profitable for you to decommission them.

[PunchyHamster]

In context of datacenter using AI workloads, it's cheaper to replace them after few years with faster, more energy efficient ones, because the power cost is major factor

[vrighter]

GPUs in your average home PC has a longer lifespan. Datacenters run them at full load for very long periods of time. Some datacenters literally burn through hundreds of GPUs a day.

[naasking]

> A bridge with a 6 year lifespan for each beam is insane.

Not necessarily. Depends entirely on the value of the transport that the bridge enables.

[jiggawatts]

> retain that capbibilty forever

Not really. The base training data cutoff will quickly render models useless as they fail to keep up with developments.

Translating some Farsi news articles about the war was hilarious, Gemini Pro got into a panic. ChatGPT either accused me of spreading fake news, or assumed this was some sort of fantasy scenario.

[jeremyjh]

Karpathy - and others - consider the pre-training knowledge as much a liability as an asset. If we could just retain the emergent reasoning and language capability without the hazy recollections the models would likely be stronger.

[m00x]

That's GPT4 thinking. New models use tools to look at current events or latest versions, and rely very little on weight knowledge.

[zozbot234]

You can pull new information into the context via RAG, but that is expensive and only gives very shallow understanding compared to retraining.

[nl]

Not really.

For coding I care mostly about reasoning ability which is uncorrelated with cut off

[loandbehold]

You need to separate training and inference usage of GPUs for this analysis.

[Mathnerd314]

"Inference consumes 60–90% of total AI lifecycle costs." So shovel is not the right analogy, more like GPU = coal burning engine. And yes, coal was a big railroad expense, more so than financing construction debt.