[nnutter]

They measured long enough in this specific example that they avoided the problem but something I see people repeated forget is to establish a baseline before making a change. If you don't know how variable something is before you make your change you might naively think you made something better or worse when it's within normal variability.

[ecaml]

Good point, and measuring things as they are before experimenting can also serve as a valid motivation to start an experiment.