The only thing SPDK buys you is somewhat lower latency, which isn't that important for most applications because modern high-performance I/O schedulers usually are not that latency sensitive anyway.
The downside of SPDK is that it is unreasonably painful to use in most contexts. When it was introduced there were few options for doing high-performance storage I/O but a lot has changed since then. I know many people that have tested SPDK in storage engines, myself included, but none that decided the juice was worth the squeeze.
OFC, If your block size is large enough and/or your design is batching enough etc. that you already don't spend much time in issuing IO/reaping completion then as you say, SPDK will not provide much of a gain.
With something like ceph, latency is everything as writes need to be synchronously committed to each OSD replica before the writing client is unblocked. I think for ceph they are trying to move to nvme-of to basically bypass the OS for remote NVME access. Im not sure how this will work security wise however as you cannot just have any node on the network reading and writing random blocks of nvme-of devices.
Not an expert on this but I think that's... half-true? There is namespace support which should allow multiple users I think (?), but it does still require direct device access.
Namespaces are a hack device manufacturers came up with to try to make this work anyway. Namespaces at the device level are a terrible idea IMP because it’s still not multi tenant - your just carving up a single drive into logically separated chunks that you have to decide on up front. So you have to say “application X gets Y% of the drive while application A gets B%”. It’s an expensive static allocation that’s not self adjusting based on actual dynamic usage.
Dynamic allocation implies the ability to shrink as well as grow. How do you envision shrinking an allocation of blocks to which your tenant has already written data that is (naturally) expected to be durable in perpetuity?
you mean something filesystems do as a matter of course? Ignoring resizing them which is also supported through supporting technologies I’m not talking about partitioning a drive. You can have different applications sharing a filesystem just fine, with each application growing how much space it uses naturally as usage increases or shrinks. Partitioning and namespaces are similar (namespaces are significantly more static) in that you have to make decisions about the future really early vs a normal file on a filesystem growing over time.
If you're assuming that every tenant's block device is storing a filesystem, then you're not providing your tenant a block device, you're providing your tenant a filesystem. And if you're providing them a filesystem, then you should use something like LVM for dynamic partitioning.
The point of NVMe namespaces is to partition at the block device layer. To turn one physical block device into multiple logical block devices, each with their own queues, LBA space, etc. It's for when your tenants are interacting with the block device directly. That's not a hack, that's intended functionality.
The downside of SPDK is that it is unreasonably painful to use in most contexts. When it was introduced there were few options for doing high-performance storage I/O but a lot has changed since then. I know many people that have tested SPDK in storage engines, myself included, but none that decided the juice was worth the squeeze.