[GauntletWizard]

I would guess your problem is probably scheduler based. The default(ish) Linux scheduler operates in 100ms increments, the first use of a client takes 3-4 round-trips. TCP opens, block, request is sent, the client blocks on write, the client attempts to read and blocks on read. If CPU usage is high momentarily, each of these yields to another process and your client isn't scheduled for another 100ms

[jontonsoup]

Hmm. We have super low CPU utilization- something like 9%. This is also across 10+ different clusters.

[jontonsoup]

We also pool our clients heavily. Maybe we could reduce the new connections to zero to test.