[finnh]

"Tell me the 99th percentile latency. Track it. Improve it. When needed, throughput comes easy."

Actually, if you're worried about tail latency, it can be improved via throughput. Expensively:

Double your entire infrastructure. Send each request twice, once to each half. Take the results of the faster one.

I assume that Google does something like this for fast search results.

[packetslave]

Yep, and you don't even (necessarily) need to double your infrastructure to do it. Jeff Dean talks about some strategies in "The Tail at Scale"

http://research.google.com/pubs/pub40801.html (paper)

http://www.youtube.com/watch?v=C_PxVdQmfpk (talk)