[leoqa]

I have 3 3090s, all in various gaming computers. I’m not sure what to do with them- the VRAM is only 24GB so doesn’t seem worth the power costs to run an open weight model. Any suggestions on how I can run them as some type of cluster? Is it worth exploring NVLink?

[HenryMulligan]

One RTX 3090 would allow you to run either Qwen 3.6 model with ~100k context at Q4_K_M quantization. Two would definitely allow full context and then some, or go for Q8 quantization. You can do quote a lot of LLM stuff with one or more 3090s. Or give them to me and I will happily put them to use!

[bigyabai]

You can layer models across multiple GPUs for inference. If you pooled it into one system, that would be 72gb of some pretty damn fast memory. Would be great for dense models in the 30-50b range.

[SyzygyRhythm]

As others said, you can run more than one at a time, but even singly, 24 GB can be very useful in a hybrid approach. For instance, I recently had to process hundreds of high-res blueprints. Used Fable for the main orchestration, but all of the OCR and other work I did with a small local model. Didn't waste expensive Fable tokens on simple grunt work.

[0x6c6f6c]

What are you using for OCR locally? This is something I've been considering trying for some time with old journals.

[fragmede]

You can put dual 3090's into one system.

https://www.reddit.com/r/LocalLLaMA/comments/1m5fkts/my_prac...

[sixothree]

Club 3090 will make installing llm easier, for 1 or more cards.