[fooker]

H100 is nearing five years and costs more to buy a used one now than a new one when it was released :)

You are completely missing the bet these companies are making.

They think can outlast their competitors and capture a larger portion of the pie while the cost of inference keeps going down dramatically.

If you haven't been paying attention, the cost is about 1/100th of what it was in 2024. This is the trajectory pretty much every technology has followed.

Of course there will be market crashes and corrections and things like that and most companies won't survive, but the bet is that whoever survives ends up doing pretty well.

[maxglute]

The fact that supply constrained GPUs holding value is negative indicator. It's like the tulips mania except bubble enthusiasts are buying wilted/dead tulips because live tulips supply constrained due to irrational demand. Now GPUs has more gross utility than tulips but seems like at current revenue/capex spend, every GPU is still negative net financial yield - they lose money - literally buying tulips and watching it wilt. Economically, better off simply not buying and losing more. That's the level of economic irrationality at place sustaining bubble, at least for hyperscaler/big tech balance sheet - small operators logic different and antagonistic to big operator demand/business model.

If cost of inference goes down 100x, would need 100x more demand. This makes overspending on GPU even more irrational. Jevons this, Jevons that but ultimately irrelevant. At end of day, leading players, hungergame winner candidates is saddling themselves with so much debt, even if they survive, post crash they are immediately uncompetitive against new entrant with blank slate and newer gen, more efficient GPUs that will be cheaper to buy/operate post crash when hardware prices will revert to mean.

It doesn't matter if some of the current players survive, they've basically stabbed and weakened themselves so much any healthy upstart in the future can wipe them out unless they lock in legislative protection... safety regulations, ban open source models etc.

That is the new bet, regulatory capture moat, because economic bet is entirely lost, especially with open models eroding mote.

[fooker]

You're making a classic philosophical error here by accidentally anthropomorphizing companies :)

A company losing out on a risky bet and failing, letting a new upstart rise up is pretty natural. A large fraction of experts from the failed companies continue at the new ones, business as usual. There are engineering teams at $BIGTECH now full of OS, database, or compiler experts from XP, Sun, HP etc.

This natural ability of companies to take risky bets is what made silicon valley successful.

[maxglute]

Companies are run by people, 100 billion dollar companies are ran by people who want to stay billionaires and will street accordingly. But I'm not sure what we're disagreeing on, yes the talent will migrate, AI industry will eventually settle on some none bubble equilibrium, but that doesn't mean current AI economics is sensible, or inflated hardware costs beyond yield is not danger indicator. Like yeah, risky bets are burning, most people are going to move and carry their technical expertise on instead of unalive themselves or flip burgers, but that's independent of whether business models and broader economy is going to explode.

[fooker]

> that's independent of whether business models and broader economy is going to explode.

It's independent of the former, but not the later. That is my point.

Businesses and business models fail all the time, does not mean the 'broader economy is going to explode'.

It could, sure. But that has been predicted several hundred times and happened only a few times.

[maxglute]

You're making a classic logic error by accidentally equating corner bodega with trillion dollar keystone industry.

Business models fail all the time, trillion business models that underpins entire macro economic growth engine exploding and causing massive contagion happens rarely, not even generationally. But when they do the leading bubble mathematic indicators are more clear.

[redwood]

You may be conflating magnitude of dollars invested in AI being at records with the fact that they're actually a lower percentage of GDP than other speculative bubbles have been ... this is important to understand because the magnitude of a slow down maybe much smaller than you seem to think

[Imustaskforhelp]

> They think can outlast their competitors and capture a larger portion of the pie while the cost of inference keeps going down dramatically.

We are also within an arms race of training newer larger models with more speed while discontinuing older models.

Gemini/Chatgpt have already discontinued their models from 2024 (iirc) because they are using all their compute in serving/training newer models. Being quite frank, nobody is serving a model from 2024 as the intended use-case while having very little moat as open source models are catching up.

> Of course there will be market crashes and corrections and things like that and most companies won't survive, but the bet is that whoever survives ends up doing pretty well.

How so, by raising the prices? because the current prices aren't sustainable and I feel as if there would certainly be companies which will try for one reason or other to be cheaper to capture the market share because of the larger promise of whoever is able to get as market share. I had once thought about it and I don't think that even in an ideal world, they would end up doing pretty well given no moat.

Also even if a company survives and ends up being one of the survivors and makes profit in the ideal scenario you mention, then within some years other companies will try again and construct more datacenters and end up driving the prices down for everyone, so nobody knows how things might look down for 2-3 years let alone a decade, so I remain a bit skeptic currently so.

I had actually thought some on the economics of datacenters and I found it to be very related to power. The only ones which seems to be making money might be the power generators actually because power is the actual bottleneck rather than GPU's in datacenters from my understanding.

Though the power is raised at the cost of electricity bill increases for everybody including people living in houses. The job prospects are minimal as well, as a nation, aside from just getting investment just for the sake of it because AI's trendy right now, I feel like its a net negative deal for people living there.

[khurs]

>H100 is nearing five years and costs more to buy a used one now than a new one when it was released :)

Because everyone is buying as they want to run their own models and not pay for a cloud service?

[mattnewton]

Because demand for inference tokens is above supply

[fooker]

Because there's no supply, data centers with these GPUs are running reasonably well.

[etempleton]

This is absolutely the calculus. There is no moat. It is survival of the best financed. Open AI and Anthropic are in very precarious situations.

[justsid]

Isn’t that the same bet that famously profitable companies like Uber did in the ride share market?

[fooker]

It's not the exact same bet, but it rhymes.

It worked out for Uber, they are wildly profitable now after spending a decade losing money.

Also worked out as Amazon managed to outlast all the dotcom era e-commerce competitors while being unprofitable.

[disgruntledphd2]

> It worked out for Uber, they are wildly profitable now after spending a decade losing money.

Uber have a 10% margin, which is definitely not what I'd consider wildly profitable. (Their post tax numbers look better, because of accumulated losses).

[Imustaskforhelp]

Uber is a two sided marketplace and was still famously unprofitable for at the same time.

Over 14 years, Uber burnt ~31 Billion dollars and in nearly whereas the amount invested within AI seems to be within Trillions at this point and in near future with a product which doesn't have much moat and shaky financials on profit.

[boesboes]

What costs are 1/100th?

[fooker]

Of serving a (approximately) gpt4 sized model.

[usrusr]

What made the cost go down? Can't be cheaper used H100, can't be cheaper RAM. A revolutionary breakthrough in hardware use per query?

[barumrho]

Is this true? Hardware costs have only gone up during this time. Are you referring to electricity cost to serve these models? (i.e. compute got more efficient?)

[underlipton]

Does that include the capital costs of spinning up to the current models/scale or is it just running costs?

Also, lost revenue from other services being degraded by shifting resources to supporting training/serving models (Google Search...)?

[alangibson]

So the number is irrelevant. No one wants yesterdays newspaper.

The only relevant number is the price to serve a frontier or near-frontier model.

[lardosaurusrex]

You just stated yourself that it costs more now used than when they were new.

If everyone's running local then why are these larger companies dumping cash into data centres?

[fooker]

Economies of scale.

You need a cluster of 8-12 H100s to run the largest models locally.

It doesn't make sense to run these locally yet unless your use case also involves making it available for several dozen concurrent users.

[kingleopold]

not to miss, future models will be more compute hungry too. Current hardware prices are still goin up and no it's not cheaper to run your AI for like %99 of the people because of lots of costs, it's not just hardware.

[ericd]

The per token costs plummet with more concurrents. A box that can do 100 tps at request depth 1 might be able to do 3000 tps at request depth 64. Less per thread, but massively more per GPU/joule/etc. That’s the economy of scale of running in a DC rather than locally that they were referring to.

[lardosaurusrex]

I think this fails to take into account how many people are fine with "fast enough" vs "fastest".

I've seen people happily use AI that takes several minutes to generate text or edit an image because to them they already aren't using their computer when they tell it to start; they just grab their phone and walk away and come back only to check in on it.

I feel like people here and on other technology discussions -- although it's worse here -- don't seem to parse what being the minority means.

They know they're one of the few to have access to such incredible hardware -- whether it be rented or purchased for way too much cash -- but they only see their own kin; their own ilk. They only compare themselves to the best.

The reality is that nobody expects data centre speed nor power in their own home and are satisfied to just go "haha its thinking" and let their computer quietly tick in the background as opposed to paying outragious prices for subscriptions or hardware.

[gmadsen]

That is completely discounting future capabilities and new use cases. Sure in 10 years you will have current SOTA locally, but in no way is it obvious we are anywhere near the limit of marginal value from improved capability

[lardosaurusrex]

Unless you're a developer or doing complicated research you do not need much to use an LLM at home with consumer-grade hardware.

I'm editing photos with 32GB of DDR4 RAM at 3200mhz alongside a 3050 with a nearly decade-old mobo and ryzen 5800XT and at most? It takes 30 seconds and that's allowing the gpu to use my actual RAM as 'fallback' memory.

There are photoshop filters -now- that take longer than that before the LLM craze even began. Hell; I can train a lora in an hour or two.

This obsession with "more specs more data faster and faster" isn't going to win; it already isn't winning.

Deal with it or get wrecked.

[spwa4]

Google's doing a attempt to answer that (while still firmly hiding who their customers are) here: https://blog.google/innovation-and-ai/technology/research/un...

They promise updates.

[xnx]

> If everyone's running local

Who's running local? Image generation can make sense to run locally, but frontier LLM make no sense to run on your own hardware.