[virtuallynathan]

With Mellanox ethernet switches, you lose some other features in exchange for lower latency. i.e. you can only break out 16 ports to 4x25G on a 100G switch (64 25G ports).

[OldManAndTheCpp]

And you also get some benefits: enough internal bandwidth to allow every input port to cut through at line rate, and a single buffer to prevent starvation (the tomahawk chip has 4 groups of ports, each with a separate buffer).

Also, you can run cumulus on the switch, which is pretty awesome.