[biddit]

That ~119 GB optimization is great. Hadn't seen that yet. That pops a 4x-Spark setup to 476 GB, which gives a lot more headroom for running something like GLM 5.2 at 4-bit, which isn't completely terrible.

I should have prefaced my post - I almost bought four Sparks a couple months ago, but ultimately opted to buy two more RTX 6000 Pro Max-Q's.

It was a painful choice because the two 6000's were more expensive than four Sparks, and ultimately gave me only 384 GB VRAM.

It was even more painful when GLM 5.2 was released, and a 4x Spark setup could run it at a decent quant, but 4x 6000's cannot with any headroom.

But the 6k's absolutely destroy the Sparks on prefill and inference speed. Model intelligence is compressing. The smaller VRAM pool will matter less over time than slower prefill/inference speed.

That is to say, I'm sure we'll end up with <500B parameter models that are Fable-level in the next 8 months or so. Performant quants of those will fit comfortably in 384 GB.

[adrian_b]

In your previous posting you said that for more than 2 Sparks you also need to buy a very expensive switch.

That is not really true. Once you have 2 fast Ethernet ports, like DGX Spark has, you can interconnect any number of systems without using a switch.

In the simplest case, you just daisy chain the systems and you configure in Linux the Ethernet interfaces as bridges.

For better performance, you can close the chain into a ring, with an extra cable. In this case the Linux configuration is more complex, because you must do IP-level routing, preferably with a routing protocol like OSPF, to be able to double the throughput between 2 systems, by using both paths through the ring.

The only advantage of a switch is a greater throughput when there are 4 or more systems, which happens only when all the interconnected systems are attempting to communicate simultaneously, in which case having only 2 paths through a ring will serialize some of the transmissions.

When only 2 systems attempt to communicate with each other, a ring has double throughput in comparison with a switch. You need 2 switches to match the throughput that a ring has, as long as it does not become congested. Or you could use one double-sized switch, with the ports partitioned between 2 VLANs, with each DGX Spark connected to 2 ports, in different VLANs. Both 2 switches or a double-sized switch would greatly increase the price.

For only 3 systems, a switch is useless, as it is worse than connecting the systems in a triangle and configuring the IP addresses for point-to-point links (the configuration as bridges is needed only for 4 or more daisy-chained systems). The configuration with 3 DGX Sparks seems optimal from the PoV of the performance per dollar ratio.

[gerdesj]

On each Spark you can bridge the two interfaces but you cannot bridge those bridges at layer two and it sounds from your post that you are confusing layer two and three (ARPA).

So if you have three Sparks, you have three two port switches. Take three two port switches and connect them in a ring and you have a collision storm. You have to use STP or similar to sort that out.

The ring is now collapsed to A-B-C (with no A-C) and to get from A to C you have to go via B which halves thoughput for those two paths: A-C and C-A.

Another "option" is to define two VLANs and use MSTP and layer three and hope the protocol that shuffles data can route packets effectively. So you get two sets of paths at layer two: A1-B1-C1-A1 and A2-B2-C2-A2 and you cut the two graphs at different points, so you get two disjoint graphs, for example: A1-B1-C1 and B2-C2-A2. However they will still overlap and it all falls apart, somewhat.

You can only link two Sparks together without a switch to get the full possible throughput. That is why Nvidia only offer a two pack option. Three or more requires a switch to avoid a path being used twice. Ethernet is not Token Ring! Mind you Tring was 16Mb/s - luxury when ethernet was 10Mb/s 8)

I've also had this discussion with folk when it comes to "hyper converged" virty systems: So we have three nodes with two NICs each for Ceph and we cable them up in a ring and it goes weird ... . At least there I'm only generally specifying 10G.

Now, do you really need the full 200Mb/s for clustering SParks? No idea yet, I've only got one and I've only got 40Gb/s QSFP+ to play with on my switches.

However I'm seeing value in the one box already at the moment and if that scales then I'll be buying more Sparks and a really funky switch when I need three Sparks and perhaps another switch down the line with some really fancy VLT or whatever wankery is the stacking du jour thing!