[jsn]

From what I can see, sendfile isn't really async, it still returns the number of bytes written or error status, and nginx probably needs to wait for that to e.g. log the results, handle short writes, or proceed with whatever next operation must be done to the socket (like closing it). No idea why aio_write needs offloading, though.

[taf2]

They say why in the article

"The asynchronous interface requires the O_DIRECT flag to be set on the file descriptor, which means that any access to the file will bypass the cache in memory and increase load on the hard disks"

They even go into how no interface has been surfaced yet that allows determining whether a file is in cache or not. Also how FreeBSD's aio interface doesn't have the same limitations

[dom0]

I never tried, but mmap and mincore could work (or not, as one of the many special cases in this area). I expect that's not a viable approach for this application, anyway :)