[dadkins]

Jim Roskind had a really nice talk [https://www.youtube.com/watch?v=_uaaCiyJCFA] at AWS Reinvent 2022 which highlighted the fence post problem. Basically, AWS was targeting P50 and P90 latencies in many services, and sure enough there were big clumps of latencies right under those target fence posts as engineers were basically gaming the metrics and nerfing the >P90 latencies to make it happen.

[fc417fc802]

I don't think that counts as gaming the metric. It's an example of a metric that even when targeted in good faith results in consequences management didn't intend.

[TeMPOraL]

Good faith starts with understanding that "90" in P90 is an arbitrary number, and that the exact sampling threshold is not important to why anyone cares about that metric.

[khuey]

The 99 in his TM99 is also an arbitrary number.

The more interesting part of the talk is how using what he calls "fenceposts" naturally results in bunching up against the fenceposts because even small regressions that cross the fenceposts are blocked but regressions of any size that don't cross a fencepost go unnoticed.

[fc417fc802]

It's just a plain bad metric. Useful due to brevity though.

A decent metric for testing against might consist of a target mean, the sigma specifying a gaussian about that mean, a minimum sample size, and a hard upper cutoff.