[bbjnicklin]

Glastra. Maybe this data is a bit better for you, direct from fio.

  TLSv1.2: ecdhe-rsa-aes128-gcm-sha256
  
  QD1: (groupid=0, jobs=1): err= 0: pid=2140: Sat Mar  5 14:59:49 2016
    read : io=2308.5MB, bw=78791KB/s, iops=19697, runt= 30001msec
      slat (usec): min=2, max=21, avg= 2.42, stdev= 0.50
      clat (usec): min=39, max=56179, avg=47.66, stdev=126.45
       lat (usec): min=46, max=56182, avg=50.14, stdev=126.45
      clat percentiles (usec):
       |  1.00th=[   46],  5.00th=[   46], 10.00th=[   46], 20.00th=[   47],
       | 30.00th=[   47], 40.00th=[   47], 50.00th=[   47], 60.00th=[   47],
       | 70.00th=[   48], 80.00th=[   48], 90.00th=[   49], 95.00th=[   49],
       | 99.00th=[   51], 99.50th=[   51], 99.90th=[   53], 99.95th=[   56],
  
  
  IPsec: aes128-gcm96
  
  QD1: (groupid=0, jobs=1): err= 0: pid=2442: Sat Mar  5 15:27:04 2016
    read : io=1902.6MB, bw=64938KB/s, iops=16234, runt= 30001msec
      slat (usec): min=2, max=29, avg= 2.39, stdev= 0.52
      clat (usec): min=52, max=57367, avg=58.51, stdev=140.69
       lat (usec): min=57, max=57369, avg=60.97, stdev=140.69
      clat percentiles (usec):
       |  1.00th=[   56],  5.00th=[   56], 10.00th=[   57], 20.00th=[   57],
       | 30.00th=[   57], 40.00th=[   57], 50.00th=[   58], 60.00th=[   58],
       | 70.00th=[   58], 80.00th=[   59], 90.00th=[   60], 95.00th=[   61],
       | 99.00th=[   63], 99.50th=[   65], 99.90th=[   78], 99.95th=[   82],

[feld]

latency is especially important for small I/Os, but there are a ton of things that can be done to improve latency in general at an OS and network level

[bbjnicklin]

Feld, you are totally right. In fact, there has been discussion in the Linux community about improving IPsec performance. However, as the post states, the difficulty in doing so is the API contract in the kernel. That said, even with a perfectly optimized IPsec stack, having to process each packet individually will always be slower than processing a logical record that is a multiple of packet size.