[anonymousiam]

I've done a lot of headless/diskless stuff. I haven't done much for years, because my NAS only has gigabit Ethernet ports. I can cascade them and get four Gbps downstream, but it's still painful.

I have recently upgraded my house to 10Gbps Ethernet, with only one room still stuck at gigabit, and unfortunately, it's my main office. I'm working on getting the drop there now (literally, just taking a break here).

Even once I'm done, accessing an iSCSI drive over 10GbE will be 4-8 times slower than a local NVMe drive, but it will sure be a lot better than it was!

Ideally, I could run VMs on the NAS and have great performance, but that's another hardware upgrade...

[olavgg]

Using a proper NIC (Chelsio) with their iSCSI accelerator will boost your iSCSI performance significantly. Another alternative is Mellanox with RDMA. You need CX4+ for optimal performance over TCP/IP, while the cheap CX3 is excellent with IPoIB. If you have a lot of packet drops and retransmissions, another option for boosting iSCSI performance is getting a network switch with a lot of memory for packet buffering. This helps with incast congestion. There are special switches with gigabytes of memory built for this.

NVMe-oF is the best protocol with least overhead for network drives, with a proper setup you lose only 10-20% latency compared to local disk even with Intel Optane. Throughput should be almost similar.

[RulerOf]

> Another alternative is Mellanox with RDMA. You need CX4+ for optimal performance over TCP/IP, while the cheap CX3 is excellent with IPoIB.

Do these benefit the iSCSI target end of the equation too, or just the initiator? And do they work like an HBA, where you configure the card in a firmware setup menu, or does it just transparently accelerate the software initiator on Windows/Linux?

[olavgg]

It is a bit different, Mellanox with iSCSI/iSER over IB/RoCE is much less complex than iSCSI over TCP. RoCE runs over UDP, but requires switches with PFC, and ECN. Chelsio plays nice with any switch. Mellanox has much better offloading of NVMe-oF, where the network card can directly communicate with the NVMe device over pci-express so it can completely bypass system ram and host cpu. In fact, Linux will know nothing about the transfers to and from the NVMe device.

To get accelerated iSCSI, you need to install Chelsio or Mellanox drivers. While both work out of the box with the inbox drivers, they have special drivers that you have to download and install to unlock the extra performance. I think Chelsio has everything included in FreeBSD, so there the inbox drivers already come with top performance unlocked.

[ReDress]

Really I wonder how this turns out to be diskless while you're clearly accessing a disk/drive over the network. Shouldn't we refer to this as network boot?

[pdpi]

It's diskless from the point of view of the device being booted.