[bluesmoon]

Posted this comment on the article, but thought it would be useful here as well:

Good points on why average latency is a bad metric, and while the idea behind Apdex was good, it never ended up being the right measure. The Apdex score still depends on a HiPPO (Highest Paid Person''s Opinion) to determine what T should be, and this can change over time.

At SOASTA (and previously at LogNormal), we borrowed the concept of LD50 (the median lethal dose) from biology. The LD50 value has the property of adapting to what your audience thinks rather than what your HIPPO thinks is a good experience.

We described the method at the Velocity conferences (Santa Clara and London) last year, and wrote it up in a blog post here: http://www.lognormal.com/blog/2012/10/03/the-3.5s-dash-for-a...

Hope you find it interesting.

I should also mention that it's useful to apply some kind of smoothing to timeseries data (like latency over time). Holt-Winters double-exponential smoothing is particularly good at this. What it does is smooth out temporary glitches and show you when things turn unexpectedly bad. If you've ever received a page and said, "Oh yeah, that one... that goes away in 3 seconds. Happens every day.", then you'll find this useful. H-W D-E smoothing only shows you the ones that don't go away after 3 seconds.