[jeffinhat]

I've been messing with NVMe over TCP at home lately and it's pretty awesome. You can scoop up the last-gen of 10GBe/40GBe networking on eBay for cheap and build your own fast disaggregated storage on upstream Linux. The kernel based implementation saves you some context switching over other network file systems, and you can (probably) pay to play for on-NIC implementations (especially as they're getting smarter).

It seems like these solutions don't have a strong authentication/encryption-in-transit story. Are vendors building this into proprietary products or is this only being used on trusted networks? I think it'd be solid technology to leverage for container storage.

[stingraycharles]

I just use iSCSI at home but using Mellanox’ RoCE which is pretty well performing.

One thing I’m noticing is that most of these storage protocols do, in fact, assume converged Ethernet; that is, zero packet loss and proper flow control.

Is this also the case with NVMe over TCP?

[jeffinhat]

I haven't experimented with it yet, but I expect that over TCP things degrade more gracefully. It seems earlier iterations of storage over networking didn't want to pay the overhead of TCP and lost out on the general purpose benefits that it brings. IIRC some RoCE iterations aren't routable for example. In theory, you could expose your NVMe over TCP device over the internet.

It seems to me applications taking advantage of NVMe are focused on building out deep queues of operations which may smooth out issues introduced by the network. But only way to know is to benchmark.

[isotopp]

The point of TCP is to provide protection against packet loss and reordering, and to provide flow control.

[effie]

> build your own fast disaggregated storage on upstream Linux

Why is this better than just connecting the drive via PCIe bus directly to the CPU?

[jeffinhat]

It certainly isn't faster or more reliable for a single node, but this is just homelab stuff. Nothing in mine is particularly necessary. I think it's interesting because it's now accessible and receiving support from multiple vendors.

At some scale, it's nice to separate the two. You don't care much about where the disks live vs. where your compute is running. You can evict work from a node, have it reschedule, and not have to worry about replicating the data to another machine with excess capacity. Though I'm no authority on this topic.

[wmf]

If you want a whole drive, sure. But if you want to virtualize storage (e.g. split up one drive) you need a network protocol.

[effie]

I don't need a network protocol to use one drive for 100 VMs, PV drivers in Xen or VirtIO in KVM work well.