[cmccabe]

> The solution here is to figure out why your 99th is 3 seconds. Once you solve that, randomized routing won't hurt you anymore. You hit this exact same problem in a non-preemptive multi-tasking system (like gevent or golang).

The Golang runtime uses non-blocking I/O to get around this problem.

[cmccabe]

Wow, it's annoying to get modded down for saying something correct. Don't believe me? Run a golang program with strace -f sometime.

You could write a pthreads-compliant threading library without using threads at all, just epoll.