[biddit]

Please don’t buy a DGX Spark unless all three of these are true:

  - You value simplicity more than performance or price-to-performance.
  - You accept that the hardware will depreciate rapidly.
  - You’re prepared to buy two or four of them.

OR:

  - You want to run frontier models right now as cheaply as possible
  - You want to run high-parameter models on a 15a breaker/line

Otherwise, get a normal, high-bandwidth GPU.

A single Spark gives you roughly 115 GB of usable memory compared with the 24–32 GB found on many lower-cost GPUs. It's certainly a big increase, but in practice it does not unlock dramatically better models.

  - One Spark: More memory, but mostly enough for poor-quality, extremely low-bit quants of larger models.  
  - Two Sparks: Enough for mid-tier parameter models at reasonable quants, such as DeepSeek V4 Flash and HY3.
  - Four Sparks: Enough for GLM 5.2 at a reasonable quant.  You'll need a $1000+ switch too.

The problem is that Sparks are slow compared with almost everything else in their price range. Many factors affect inference speed, but memory bandwidth is one of the biggest. A $4,000-plus DGX Spark provides only 273 GB/s.

  | GPU                 | Memory bandwidth |           VRAM | Approx. price  |
  | ------------------- | ---------------: | -------------: | ------------:  |
  | DGX Spark           |         273 GB/s | ~115 GB usable |        $4,000+ |
  | RTX 5060            |         448 GB/s |          16 GB |          $600  |
  | Radeon AI Pro R9700 |         640 GB/s |          32 GB |        $1,200  |
  | RTX 4000 Pro        |         672 GB/s |          24 GB |        $2,300  |
  | RTX 4500 Pro        |         896 GB/s |          32 GB |        $3,500  |
  | RTX 3090            |         936 GB/s |          24 GB |        $1,200  |
  | RTX 5000 Pro        |       1,344 GB/s |          48 GB |        $6,000  |
  | RTX 5090            |       1,792 GB/s |          32 GB |        $4,000  |
  | RTX 6000 Pro        |       1,792 GB/s |          96 GB |       $12,000  |

Yes, the Spark has substantially more memory. But going from roughly 24 GB to 115 GB does not necessarily unlock substantially better model quality. In many cases, it only lets you load heavily compressed 2-bit versions of larger models, such as DeepSeek V4 Flash, with serious quality degradation.

24–32 GB is currently a sweet spot. Models such as Qwen 3.6 27B and 35B-A3B:

  - Perform far above what their parameter counts suggest.
  - Fit comfortably within 24–32 GB of VRAM at reasonable quantization levels.

A 4-bit quant of Qwen 3.6 27b (18 GB) will out-perform a 2-bit quant of DeepSeek v4 Flash (90gb).

Instead of the Spark, if I had a roughly $4,000 budget...

Assuming I already had a reasonably modern desktop:

  - One RTX 5090, RTX 5000 Pro, or RTX 4500 Pro.
  - Two RTX 3090s, RTX 4000 Pros, or R9700s, provided the motherboard can bifurcate two physical x16 slots into x8/x8.

If I were building a system from scratch:

  - A DDR4- or PCIe 4.0-era consumer CPU and motherboard that supports x8/x8 bifurcation.
  - Two RTX 3090s, RTX 4000 Pros, or R9700s.

If I were already planning to buy a new Mac:

  - A MacBook Pro M5 with 64 GB or 128 GB of unified memory.

For context, these are the systems I currently run:

  - EPYC Turin with four RTX 6000 Pro Max-Qs.
  - EPYC Milan with four RTX 3090s.
  - AM4 with two RTX 3090s.
  - AM4 with two RTX 3090s.
  - Intel Raptor Lake with two RTX 5060 Ti.
  - MacBook Pro M3 128GB Unified

[josh-wrale]

I agree with this. I have a dual rtx4090 machine, a 128gb m5 max mbp, and a dgx spark. RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic on the dual rtx4090 machine under vLLM absolutely slays at token speed. I'm frustrated with how hard it is to realize goodness on the dgx spark.

[pizza234]

I'm actually confused by the post - indeed why choosing the 27B dense and the 35B MoE models? Given the structure of the post I honestly have the impression that the Spark was purchased out of curiosity more than for a specific purpose (LLMs).

[InTheArena]

The 27b dense is _remarkably_ better at coding tasks.

Especially at larger quanitizations (Q4 is pretty crap).

[biddit]

Ah yeah, your reaction makes sense. In the circles I run in, there is a lot of hype around Sparks for inference, so my gut reaction is to respond with this type of warning.

I did not intend to imply that the post author was advocating that they're great for inference, as they're obviously not.

[gerdesj]

Actually, its ~119GB usable (I have one). You can shutdown a lot of unneeded services if you only use it remote which will trim a lot more fat. You can enable the RDP service if you don't want to sit in front of it but get a desktop interface.

The "shitty" network is 10Gb/s and wifi7! You get a twin QSFP28DDlol+++ (I jest) that each run at 200Gb/s - not for the casual home user but handy at work, although I "only" have 40Gb/s on my switches sigh. With and no switch two you can do a three node cluster with some careful networking. If you want to do more then a switch is needed and it will need to be pretty funky! That said you could wire them up in a circle and use VLANs and MSTP and accept less than 200Gb/s per link. You'll probably need Openvswitch and a lie down afterwards.

I'm not a fan of the Gnome desktop but it works well enough and I think the Nvidia customised Ubuntu is well thought out. You get all the complicated NVidia extras pre-installed, along with docker (full fat, not the Ubuntu one) for a fairly quick start. It includes Ubuntu Pro which is free for five systems anyway but its nice to see it pre-installed.

You can run quite decent models on this thing see: https://spark-arena.com/ Also see "DS4".

We blew abut £4000 on one and it will pay for itself in a few months. I tried pricing up an Apple thingie and the Store wouldn't offer me more than 96Gb of RAM and a delivery date in Q3 at the earliest. Our Spark rocked up next day. They seem to come in 1TB or 4TB SSD variants. 1TB is enough for me and saves a lot of cash - keep an eye on your model downloads and ruthlessly delete old experiments. docker system prune.

We went for the Asus variant that has active cooling and I stuck it in the ceiling cable tray over our computer room racks. It sits on 1½" stainless steel mesh with lots of clearance in an actively cooled environment.

[biddit]

That ~119 GB optimization is great. Hadn't seen that yet. That pops a 4x-Spark setup to 476 GB, which gives a lot more headroom for running something like GLM 5.2 at 4-bit, which isn't completely terrible.

I should have prefaced my post - I almost bought four Sparks a couple months ago, but ultimately opted to buy two more RTX 6000 Pro Max-Q's.

It was a painful choice because the two 6000's were more expensive than four Sparks, and ultimately gave me only 384 GB VRAM.

It was even more painful when GLM 5.2 was released, and a 4x Spark setup could run it at a decent quant, but 4x 6000's cannot with any headroom.

But the 6k's absolutely destroy the Sparks on prefill and inference speed. Model intelligence is compressing. The smaller VRAM pool will matter less over time than slower prefill/inference speed.

That is to say, I'm sure we'll end up with <500B parameter models that are Fable-level in the next 8 months or so. Performant quants of those will fit comfortably in 384 GB.

[adrian_b]

In your previous posting you said that for more than 2 Sparks you also need to buy a very expensive switch.

That is not really true. Once you have 2 fast Ethernet ports, like DGX Spark has, you can interconnect any number of systems without using a switch.

In the simplest case, you just daisy chain the systems and you configure in Linux the Ethernet interfaces as bridges.

For better performance, you can close the chain into a ring, with an extra cable. In this case the Linux configuration is more complex, because you must do IP-level routing, preferably with a routing protocol like OSPF, to be able to double the throughput between 2 systems, by using both paths through the ring.

The only advantage of a switch is a greater throughput when there are 4 or more systems, which happens only when all the interconnected systems are attempting to communicate simultaneously, in which case having only 2 paths through a ring will serialize some of the transmissions.

When only 2 systems attempt to communicate with each other, a ring has double throughput in comparison with a switch. You need 2 switches to match the throughput that a ring has, as long as it does not become congested. Or you could use one double-sized switch, with the ports partitioned between 2 VLANs, with each DGX Spark connected to 2 ports, in different VLANs. Both 2 switches or a double-sized switch would greatly increase the price.

For only 3 systems, a switch is useless, as it is worse than connecting the systems in a triangle and configuring the IP addresses for point-to-point links (the configuration as bridges is needed only for 4 or more daisy-chained systems). The configuration with 3 DGX Sparks seems optimal from the PoV of the performance per dollar ratio.

[gerdesj]

On each Spark you can bridge the two interfaces but you cannot bridge those bridges at layer two and it sounds from your post that you are confusing layer two and three (ARPA).

So if you have three Sparks, you have three two port switches. Take three two port switches and connect them in a ring and you have a collision storm. You have to use STP or similar to sort that out.

The ring is now collapsed to A-B-C (with no A-C) and to get from A to C you have to go via B which halves thoughput for those two paths: A-C and C-A.

Another "option" is to define two VLANs and use MSTP and layer three and hope the protocol that shuffles data can route packets effectively. So you get two sets of paths at layer two: A1-B1-C1-A1 and A2-B2-C2-A2 and you cut the two graphs at different points, so you get two disjoint graphs, for example: A1-B1-C1 and B2-C2-A2. However they will still overlap and it all falls apart, somewhat.

You can only link two Sparks together without a switch to get the full possible throughput. That is why Nvidia only offer a two pack option. Three or more requires a switch to avoid a path being used twice. Ethernet is not Token Ring! Mind you Tring was 16Mb/s - luxury when ethernet was 10Mb/s 8)

I've also had this discussion with folk when it comes to "hyper converged" virty systems: So we have three nodes with two NICs each for Ceph and we cable them up in a ring and it goes weird ... . At least there I'm only generally specifying 10G.

Now, do you really need the full 200Mb/s for clustering SParks? No idea yet, I've only got one and I've only got 40Gb/s QSFP+ to play with on my switches.

However I'm seeing value in the one box already at the moment and if that scales then I'll be buying more Sparks and a really funky switch when I need three Sparks and perhaps another switch down the line with some really fancy VLT or whatever wankery is the stacking du jour thing!