Yes. Most network switches can handle all ports at 100% utilization in both directions simultaneously.
Take for example the Mellanox SX6790 available for less than $100 on eBay. It has 36 56gbps ports. 36 * 2 * 56 = 4032gbps and it is stated to have a switching capacity of 4.032Tbps.
Edit: I guess you are asking how one would possibly sip 1TiB/s of data into a given client. You would need multiple clients spread across several switches to generate such load. Or maybe some freaky link aggregation. 10x 800gbps links for your client, plus at least 10x 800gbps links out to the servers.
Sure- I don't mean to imply that it isn't. I can absolutely see how that's inexpensive for 100gbe equipment.
That was more for the benefit of others like myself, who were wondering if "bargain" was comparative, or inexpensive enough that it might be worth buying one next time they upgraded switches. For me personally it's still an order of magnitude away from that.
Given their configuration of just 4U spread across 17 racks, there's likely a bunch of compute in the rest of the rack, and 1-2 top of rack switches like this:
And then you connect the TOR switches to higher level switches in something like a Clos distribution to get the desired bandwidth between any two nodes:
So the software infra for using multiple slots already exists and doesn't require any special config. Oh and some cards can use PCIe slots across multiple hosts. No idea why you'd want to do that, but you can.
Note that 36 port 56G switches are dirt cheap on eBay and 4tbps is good enough for most homelab use cases