[zdw]

This is somewhat tangential, but might help if other people are attempting to do similar things in the future...

A relation of mine is a pharmacist at a hospital. At his workplace, they have automated drug dispensing machines that can be used by employees to obtain medication for patients - saves a lot of time and work for dispensing normal stuff like painkillers, etc.

These machines use a statistical method to flag when an employee is pulling out more than the usual amount of medicine, as there are infrequent cases of employees selling/using it themselves.

The machines were programmed to use standard deviation for this - if what you draw is within 2 standard deviations of the mean of all users, you're fine.

The problem is that, on one occasion, an employee in a small section was a junkie and pulled out so much of a certain drug that the mean was skewed to the point that they were still within 2 std dev of it, and it wasn't noticed for a few months.

So, to wrap up the story, you probably don't want to purely use statistics to test for failure. You want some basic sanity checks in there.

[hle]

"All the models are wrong but some are useful". I think that in this case the model was just wrong... or a bit simplistic.