[9front]

Intel's E810 based NICs require only a PCIe3.1x16 slot. 16 lanes will accommodate the 100GbE port just fine. Theoretical PCIe throughput for 16 PCI lanes is around 252Gbps. The 800 NIC chipset is just four 25Gb Ethernet lanes stitched together. PCIe4 won't help this NIC much.

[walrus01]

Your throughput calculation is based on a single port 100GbE NIC working okay at full duplex bidirectional line rate in a PCI-E 3.0 x16 slot, which is true. Let's say that we budget 220-230Gbps of throughput per optical transceiver, such as if you were to max out a 100GbE link both directions with iperf3 for testing. But the intel X820 card has two ports on it, so your bandwidth needs are going to be in the range of 500 Gbps.

also, cumulative number of pci-e 3.0 or 4.0 lanes in a system is a big consideration if you want to have, for instance, four dual-port 100GbE NICs all talking to one CPU. Or some mixture like three dual-port 100GbE NICs + one or two 4-port 10GbE NICs.

Where the lower end of the Intel server CPU offerings really falls flat is not having anything close to 64 or 128 PCIE lanes at a reasonable price.

[9front]

The maximum throughput of Intel's Columbiaville NICs is 100G max. When the 2nd port is used the total bandwidth is split in half, so there will be two 50G links. The 2nd port is usually in stand-by mode when the 1st port is used as a 100G link. So, two active ports on a Intel 100G NIC does not result in 200G throughput.

[shaklee3]

This is correct. The Asics on these cards cannot do 200G, even if pcie allowed. The only one I'm aware of that can is the Mellanox cx6 dx

[shaklee3]

I don't understand why you are combining tx and RX. PCI, fiber, and the switch/nic are all full duplex, so you don't need to add the two directions together. A dual port nic, even with pcie4, is unlikely to be able to hit 200G ( single direction) unless it's using very large packet sizes.

Since you're limited by the 100G negotiated speed, you nic will never send more than 100G with iperf on a single port.