[ftufek]

Local workstation is much cheaper in the long run.

Even ignoring that, most of the development is running experiments. You're gonna be hesitant to run lots of experiments if they each cost money whereas when you pay upfront for the hardware, you're gonna have the incentive to fully utilize it with lots of experiments.

I'd go with rtx 4090 and deal with memory limitation through software tricks. It's an underrated card that's as performant as cards that are magnitude pricier. It's great way to get started with that budget.

[biddit]

I agree with you but right now RTX 4090 cards are pushing $2000, which doesn't leave much budget left. I'd suggest picking up a used 3090 card from eBay, which are currently around $800. This will still give 24gb of VRAM like the 4090.

[RockyMcNuts]

i've seen some blog posts saying if you buy a used 3090 that has been used for bitcoin mining then there is a risk of thermal throttling because the thermal paste on the vram is not great and worse if it was run hot for a long time.

any recommendations on how to buy one? e.g. 24GB model, any particular model to run LLMs? what is the biggest baddest LLM you can run on a single card?

have been thinking about it but was sticking with cloud/colab for experiments so far.

[dharmab]

The good deals are gonna be on local ads. Facebook Marketplace in most of the US.

[fragmede]

Craigslist and eBay have some great deals.

[two_in_one]

I remember videos (on youtube likely) of thermal paste replacement, that was upgrade to stock card. So, average person should be able to do it. It'll cost a few $$ for the paste. I would go with local workstation, then don't have to think much about while running stable diffusion. Plus, if it's used from ebay, prices cannot go much lower, you'll get something back at the end. Also, for image things training dataset can be quite big for network transfers.

[icelancer]

Strong endorse here. I pick up used RTX 3090s from Facebook Marketplace and eBay at $800 maximum. Can usually find them locally for $700-750, and typically can test them too, which is fine (though I've had no issues yet).

[Philpax]

Depending on what you're doing, 2x used 3090s are the same price and offer you more VRAM. That's what I'm planning on doing, in any case - being able to run 70B LLMs entirely on the GPU is more useful than being able to run 34B faster.

[icelancer]

Yeah multiple 3090s is the best budget way to go for sure. Also older server boards with tons of PCIe lanes if you can swing rack mounted hardware and have some technical skills.

[biddit]

Agreed. I recently completed a new build with two 3090 GPUs and really appreciate being able to run 70b models.

[Dowwie]

which cpu did you go with?

[biddit]

i7-14700k

z790 chipset w/ mobo that supports x8/x8 bifurcation

96gb ddr5 @5600mhz