[trentor]

I'm genuinely disappointed with my Spark. I don't know how anyone can claim it performs decently with LLMs or diffusion models. Back when I worked in VFX in the early 2000s, we had a saying: "Render time is coffee time" and if you try to run this thing with a usable context size, you'll be drinking a lot of coffee. Most of the optimizations it relies on for inference simply aren't available for training, so it crawls like a snail on almost every model. An RTX 6000 Blackwell would have been the better investment for an AI enthusiasts and for general computing there are cheaper offerings.

[mapontosevenths]

If you bought it for inference you made a mistake. They aren't good at that. Use it to train models and experiment with ML. It's much better at that.

If you just want local inference buy a Mac.

If you bought early on, like I did, the Spark is probably worth double what you payed now. I think I paid $3,000 retail for mine and the last time I looked they were fetching close to $6k on ebay. I'm not sure if that's still the case, but you can buy a very nice Mac with $6k.

[trentor]

Mhm... maybe read my full comment?

[mapontosevenths]

Are you saying that it's slow for training? Sorry, your comment is confusingly worded to me.

I've not had any issues in that regard, but I'm working with LLM's not training diffusion models. Are you following one of the Nvidia provided recipes or inventing something on your own? The last time I looked into it they benchmarked very well, but we both know that doesn't always mean much.

[trentor]

Yes it is slow for training/fine-tuning decently sized LLMs and unusable for diffusion models. What are you training were you get usable speeds?

[Foobar8568]

I have access to both a RTX 5090 PC with 64gb of ram and a Spark 128gb, the performance of the Spark has been highly disappointing.

I prefer 1000x the RTX one, even with 64gb of ram.

[dllu]

The DGX Spark hits a sweet spot for me where it can simultaneously work for general computing and run local LLM inference fast enough for some hobbyist dabbling. The RTX 6000 Pro Blackwell is more than twice the price (it has increased quite a bit recently from $8000 to $11600), not to mention the "rest of the PC" needed to get it working, so it's not really a fair comparison. Compared to other 128 GB unified memory devices like the Mac Studio and the Strix Halo, the DGX Spark fairly priced in my opinion.

[icedchai]

I have a AMD Strix Halo box I use for similar dabbling. It definitely wasn't an "out of the box" experience, fiddling around with kernel versions and ROCm installs. These days I mostly wind up using the Vulkan build of llama.cpp for inference.