[NavinF]

Linked article says they used 68 machines with 2 x 100GbE Mellanox ConnectX-6 cards. So any 100G pizza box switches should work.

Note that 36 port 56G switches are dirt cheap on eBay and 4tbps is good enough for most homelab use cases

[riku_iki]

> So any 100G pizza box switches should work.

but will it be able to handle combined TB/s traffic?

[aaronax]

Yes. Most network switches can handle all ports at 100% utilization in both directions simultaneously.

Take for example the Mellanox SX6790 available for less than $100 on eBay. It has 36 56gbps ports. 36 * 2 * 56 = 4032gbps and it is stated to have a switching capacity of 4.032Tbps.

Edit: I guess you are asking how one would possibly sip 1TiB/s of data into a given client. You would need multiple clients spread across several switches to generate such load. Or maybe some freaky link aggregation. 10x 800gbps links for your client, plus at least 10x 800gbps links out to the servers.

[bombcar]

Even the bargain Mikrotik can do 1.2Tbps https://mikrotik.com/product/crs518_16xs_2xq

[margalabargala]

For those curious, a "bargain" on a 100gbps switch means about $1350

[epistasis]

On a cluster with more than $1M of NVMe disks, that does actually seem like a bargain.

(Note that the linked MikroTik switch only has 100gbe on a few ports, and wouldn't really classify as a full 100gbe switch to most people)

[margalabargala]

Sure- I don't mean to imply that it isn't. I can absolutely see how that's inexpensive for 100gbe equipment.

That was more for the benefit of others like myself, who were wondering if "bargain" was comparative, or inexpensive enough that it might be worth buying one next time they upgraded switches. For me personally it's still an order of magnitude away from that.

[bombcar]

https://mikrotik.com/product/crs305_1g_4s_in is the sweet spot right now for home users. Four 10g ports and a 1g, you can use the 1g for “uplink” to the internet and one of the 10g for your “big old Nortel gigabit switch with 10g uplink” and one for your Mac and two for your NAS and VM server. ;)

Direct cables are moderately cheap, and modules for 10g Ethernet aren’t insanely expensive.

[Palomides]

there's usually some used dx010 (32x100gbe) on ebay for less than $500

the cheapest new 100gbe switch I know of is the mikrotik CRS504-4XQ-IN (4x100gbe, around $650)

[riku_iki]

TB != Tb..

[baq]

any switch which can't handle full load on all ports isn't worthy of the name 'switch', it's more like 'toy network appliance'

[birdman3131]

I will forever be scarred by the "Gigabit" switches of old that were 2 gigabit ports and 22 100mb ports. Coworker bought it missing the nuance.

[bombcar]

Still happens, gotta see if the top speed mentioned is an uplink or normal ports.