[wkat4242]

> And who knows, maybe someone will look back on my work and be like “haha, remember when we thought 192GB of VRAM was a lot?”

I wonder if this will happen. It's already really hard to buy big HDDs for my NAS because nobody buys external drives anymore. So the pricing has gone up a lot for the prosumer.

I expect something similar to happen to AI. The big cloud parties are all big leaders on LLMs and their goal is to keep us beholden to their cloud service. Cheap home hardware work serious capability is not something they're interested in. They want to keep it out of our reach so we can pay them rent and they can mine our data.

[Eisenstein]

It isn't that cloud providers want to shut us out, it is that nVidia wants to relegate AI capable cards to the high end enterprise tier. So far in 2024 they have made $10.44b in revenue from the gaming market, and over $47.5b in the datacenter market, and I would bet that there is much less profit in gaming. In order to keep the market segmented they stopped putting nvlink on gaming cards and have capped VRAM at 24GB for the highest end GPUs (3090 and 4090) and it doesn't look much better for the upcoming 5090. I don't blame them, they are a profit-maximizing corporation after all, but if anything is to be done about making large AI models practical for hobbyists, start with nVidia.

That said, I really don't think that the way forward for hobbyists is maxing VRAM. Small models are becoming much more capable and accelerators are a possibility, and there may not be a need for a person to run a 70billion parameter model in memory at all when there are MoEs like Mixtral and small capable models like phi.

[Saris]

>It's already really hard to buy big HDDs for my NAS because nobody buys external drives anymore. So the pricing has gone up a lot for the prosumer.

I buy refurb/used enterprise drives for that reason, generally around $12 per TB for the recent larger drives. And around $6 per TB for smaller drives. You just need an SAS interface but that's not difficult or expensive.

IE; 25TB for $320, or 12TB for $80.

[thelastparadise]

> It's already really hard to buy big HDDs for my NAS

IME 20tb drives are easy to find.

I don't think the clouds have access to bigger drives or anything.

Similarly, we can buy 8x A100s, they're just fundamentally expensive whether you're a business or not.

There doesn't seem to be any "wall" up like there used to be with proprietary hardware.

[wkat4242]

They are easy to find but extremely expensive. I used to pay below 200€ for a 14TB Seagate 8 years ago. That's now above 300. And the bigger ones are even more expensive.

For me these prices are prohibitive. Just like the A100s are (though those are even more so of course).

The problem is the common consumer relying on the cloud so these kind of products become niches and lose volume. Also, the cloud providers don't pay what we do for a GPU or HDD. They buy them by the ten thousands and get deep discounts. That's why the RRPs which we do pay are highly inflated.

[walterbell]

Looking at https://shucks.top and https://diskprices.com, prices do seem to be higher.

Homelab vendor in Austin, TX with periodic sales, limited volume: https://shop.digitalspaceport.com

[Dylan16807]

Well if I look at Amazon I see a couple models of external 14TB for $190, and a brand new Exos 16TB for $230. Not too bad. Though personally I get much cheaper used drives and put them in RAID for a NAS.

And they do have better sales.

[wkat4242]

Most of the cheap drives here are refurbs with questional quality. And those Exoses here are much more expensive sadly, especially if you choose only legit vendors on Amazon.

[gizmo686]

The cloud companies do not make the hardware, they buy it like the rest of us. They are just going to be almost the entirety of the market, so naturally the products will built and priced with that market in mind.

[wkat4242]

Yes and they get deep discounts which we don't. Can be 40% or more!

Of course the vendor can't make a profit with such discounts so they inflate the RRP. But we do end up paying that.

[illiac786]

That’s the main problem is a market owned by enterprise customers. Consumers don’t matter, there is zero interest is competing for them, they’re too little. The discounts is a killer for example, well have to buy from a reseller each time, who of course will pocket a good proportion of the discount because there won’t be many resellers that sell to consumers…

I have seen very large ent customers get 80% discount on hardware - it’s mind boggling that the vendor is not going bankrupt.

[donavanm]

Not specific for GPUs but I believe some of those giant and deeply discounted buys are at/below typical cost because of volume. They allow the vendor to increase their OEM/manufacturing commits, or shift bins theyre long on, to improve the rest of their sales pipeline. Similar for very large last orders or all the remaining stock of a SKU which improves cash flow and turns over inventory. Its a very very different vendor relationship with things like defect rates, yield, and “warranty” turned in to price factors.

[wkat4242]

> I have seen very large ent customers get 80% discount on hardware - it’s mind boggling that the vendor is not going bankrupt.

Yes exactly. When I see what we pay for stuff at work...

Obviously the vendors don't have 80+% margins. So what do they do? Inflate the RRP to compensate. So they can give a huge discount that sounds good on paper.

But this makes it unviable to buy for consumers that do have to pay RRP.