[olavgg]

Using a proper NIC (Chelsio) with their iSCSI accelerator will boost your iSCSI performance significantly. Another alternative is Mellanox with RDMA. You need CX4+ for optimal performance over TCP/IP, while the cheap CX3 is excellent with IPoIB. If you have a lot of packet drops and retransmissions, another option for boosting iSCSI performance is getting a network switch with a lot of memory for packet buffering. This helps with incast congestion. There are special switches with gigabytes of memory built for this.

NVMe-oF is the best protocol with least overhead for network drives, with a proper setup you lose only 10-20% latency compared to local disk even with Intel Optane. Throughput should be almost similar.

[RulerOf]

> Another alternative is Mellanox with RDMA. You need CX4+ for optimal performance over TCP/IP, while the cheap CX3 is excellent with IPoIB.

Do these benefit the iSCSI target end of the equation too, or just the initiator? And do they work like an HBA, where you configure the card in a firmware setup menu, or does it just transparently accelerate the software initiator on Windows/Linux?

[olavgg]

It is a bit different, Mellanox with iSCSI/iSER over IB/RoCE is much less complex than iSCSI over TCP. RoCE runs over UDP, but requires switches with PFC, and ECN. Chelsio plays nice with any switch. Mellanox has much better offloading of NVMe-oF, where the network card can directly communicate with the NVMe device over pci-express so it can completely bypass system ram and host cpu. In fact, Linux will know nothing about the transfers to and from the NVMe device.

To get accelerated iSCSI, you need to install Chelsio or Mellanox drivers. While both work out of the box with the inbox drivers, they have special drivers that you have to download and install to unlock the extra performance. I think Chelsio has everything included in FreeBSD, so there the inbox drivers already come with top performance unlocked.