[torinmr]

Great article, I ran into this exact problem at work today. You can also get a very similar problem if the timeseries aggregation system you're using does any pre-aggregation before you calculate the percentile - for example, if you sample your servers every 90 seconds, then any latency number it reports is likely already averaged over the requests the server received during that time period, meaning your 99th percentile number is really the latency of the 99th percentile server, not the 99th percentile request. Using latency buckets solves this problem as well, however.