[RulerOf]

> Another alternative is Mellanox with RDMA. You need CX4+ for optimal performance over TCP/IP, while the cheap CX3 is excellent with IPoIB.

Do these benefit the iSCSI target end of the equation too, or just the initiator? And do they work like an HBA, where you configure the card in a firmware setup menu, or does it just transparently accelerate the software initiator on Windows/Linux?

[olavgg]

It is a bit different, Mellanox with iSCSI/iSER over IB/RoCE is much less complex than iSCSI over TCP. RoCE runs over UDP, but requires switches with PFC, and ECN. Chelsio plays nice with any switch. Mellanox has much better offloading of NVMe-oF, where the network card can directly communicate with the NVMe device over pci-express so it can completely bypass system ram and host cpu. In fact, Linux will know nothing about the transfers to and from the NVMe device.

To get accelerated iSCSI, you need to install Chelsio or Mellanox drivers. While both work out of the box with the inbox drivers, they have special drivers that you have to download and install to unlock the extra performance. I think Chelsio has everything included in FreeBSD, so there the inbox drivers already come with top performance unlocked.