[drv]

I am an engineer working at Intel on SPDK, and I can answer any technical questions you might have.

Currently SPDK consists of a usermode NVMe (PCIe-attached SSD) driver. We will soon be releasing a usermode driver for the Intel I/OAT DMA engine (copy offload hardware) that is available on some server platforms.

[n00b101]

Will combining SPDK with the usermode driver for the Intel I/OAT DMA engine provide the necessary building blocks for a complete solution (network+storage)?

And can this be combined with PCIe-attached accelerators (e.g. Xeon Phi or GPUs)?

[drv]

The SPDK libraries are mostly storage-specific components (the I/OAT DMA engine can be used for generic copy offload, but it is particularly useful for copying between network and storage buffers). SPDK itself does not provide any network functionality.

I am not familiar enough with the Xeon Phi or GPU programming model to say for sure, but they could possibly be used to offload tasks like hashing/dedup or other storage-related functions.

[n00b101]

> they could possibly be used to offload tasks like hashing/dedup or other storage-related functions

Sorry, I was not referring to accelerating storage-related functions, I was wondering about efficient DMA copy from one PCIe device (Intel NVM storage) to another (Xeon Phi accelerator) which would be for useful many different functions, if the NVM storage device capacity is much larger than the accelerator device memory.

[drv]

Ah, I see. The I/OAT DMA copy offload is essentially equivalent to an asynchronous memcpy(), so anything addressable on the memory bus could be a source or destination (with some caveats about alignment requirements and pinned pages if copying to/from RAM).

[pm]

As someone who has never touched any of this stuff before, where might I actually find it useful in my day-to-day programming (assuming I might have an application where it would be useful)?

[drv]

The NVMe driver will only work for a fairly narrow set of uses in which the whole NVMe device(s) can be dedicated to a single application (this is because the user-space application takes control of the NVMe device directly, so the kernel driver can't simultaneously use it).

Some of the straightforward use cases would be inside network-attached storage appliances (ideally in conjunction with a user-mode network stack) or in a database (database systems already typically want to avoid any OS interference with storage access). In general, the NVMe driver can be dropped in fairly easily when existing code is using something like Linux AIO with O_DIRECT on a raw block device; the AIO programming model maps quite directly to the NVMe driver programming model (create a queue, submit I/Os, and poll for completions).

[greglindahl]

Are you going to integrate this with the Intel Omni-Path user-level library? You could probably serve a quite insane number of transactions that way.

[drv]

I don't know anything about Omni-Path, sorry. However, based on the publicly available information, it does look like a very interesting combination. One major advantage of SPDK over the traditional kernel-provided storage stack is lower latency (by avoiding interrupts and other context switches), and it would fit nicely with a low-latency network stack.