Hacker News new | ask | show | jobs
by wmf 883 days ago
I think PCIe TLP overhead and NVMe commands account for the difference between 7 and 8 GB/s.
1 comments

You are probably right. Reading some old notes of mine when I was fine-tuning PCIe bandwith on my ZFS server, I had discovered back then that a PCIe Max_Payload_Size of 256 bytes limited usable bandwidth to about 74% of the link's theoretical max. I had calculated that 512 and 1024 bytes (the maximum) would raise it to respectively about 86% and 93% (but my SATA controllers didn't support a value greater than 256.)
Mellanox recommends setting this from the default 512 to 4096 on their NICs.