[lathiat]

It won't trigger a read if you write a full 4kB page in both size and alignment but does for anything else. And then as you said, it stalls your write.

I debugged a Ceph issue being caused by this kind of behaviour except it wasn't actually mmap it was pwritev() but the call would stall and take a long time to return in that call that was expected to be async and go into the page cache.

It was caused by "unaligned" writes, e.g. non 4kB-sized-and-aligned writes which happens from Windows guests (using 512b alignment, at least by default) but not Linux guests (which use 4kB alignment, even if the disk is 4kB aligned)

It was made worse by Ceph behaviour that would write to that page, incur the penalty, immediately use madvise to tell the kernel I DONTNEED that anymore as an optimisation (because its a replica that is write-only, I don't need to read it again!), it got dropped, and a millisecond later wrote to it again and had to read it again (because that optimisation did not consider this case of unaligned writes needing to read)

Full story here: https://www.youtube.com/watch?v=_vfGcsvnn6U "In-depth technical story: Fixing I/O performance for Windows guests in OpenStack Ceph clouds"

It was at this point I understood memory alignment and why it might matter at a CPU level.