[mjn]

This was one of the more interesting realizations when I was an undergraduate writing my first research paper. We were testing latency of MIDI interfaces, and after sanity checking by looking at some of the underlying data, realized that average, or even average+stddev, was obscuring a lot of stuff. For example, note-to-note consistency is a major issue in music interfaces, often more important than absolute latency, since the spacing between notes is very important to melody perception (games often have a similar issue).

Showing the full histogram isn't a full solution either, though. Not only does using the average latency obscure the issue by boiling it down to a single scalar, but the full histogram of latencies also loses the information on note-to-note consistency! That's because a latency histogram loses sequencing information, so it doesn't distinguish between the case where you had a lot of 20ms latencies in a row followed by a lot of 50ms latencies in a row, and the case where every other message oscillated between 20ms and 50ms latencies (much worse). You can try to capture some of that information by making a histogram of adjacent-latency deltas, as one attempt. Or you can capture a different view on it by plotting latency vs. time and looking for spikes (but that can obscure less-obvious trends, and is unwieldy as a data representation if you're trying to summarize a system's behavior over a period of hours).

The paper is here, though the actual numbers are 9 years old at this point, so probably not that useful: http://www.cs.hmc.edu/~bthom/res/midi_timing/publications/IC...