“When a measure becomes a target, it ceases to be a good measure.”
The problem is that a competitive measure tends to overshadow the utility of the measure (look at anything that starts with a TPC- or the first rotation of a Tesla roadster tire).
So the biggest impact of a measure is immediately after it is introduced, because the market hasn't engineering itself to overfit to that measure.
And that is what Dan's post clearly says, that immediately after he brings up something, there is an actual change in the industry practice that happens.
> it doesn't show off creativity.
But something like Jepsen is amazingly useful because it is a creative nothing-up-my-sleeves end-run around first-party testing - actual third party testing for the product is the red-team to a QE team doing the blue-team work. The problem is that a lot of people leave that to the same vendor who is selling the product with its promises from their sales team - it is hardly an open book into the bugs open list (I work in open-source, so I have to wash my dirty laundry in public).
As a side not to this, I had a meeting in the last 2 days where someone told me that the "QE team is doing a good job, because the test have been consistently green as the release approaches".
Most folks don't naturally like to measure things. Most people view the world through their own personal lens and they categorize phenomena through their experiences. It makes it hard to "sell" the idea of measurement, since most people only see the things they're predisposed to see.
I think my tone came across as more elitist than I meant. I certainly suffer from all of these issues! I enjoy working with data and ML so I'm aware of and try to actively combat them in myself but I'd be lying if I were to say I haven't succumbed to these problems myself in the past and don't continue to do so.
Additionally it's hard work. Creating the right tests, getting the data into the format you need to do analysis, collating it and finding the important things, explaining it.
Very hard work and often takes a lot of creativity. I always thought measurement was the essence of engineering but these days find it to be a lost art. Too often the market for the measure takes priority over the scientific piece.
Goodhart's law has been a problem with measures.
“When a measure becomes a target, it ceases to be a good measure.”
The problem is that a competitive measure tends to overshadow the utility of the measure (look at anything that starts with a TPC- or the first rotation of a Tesla roadster tire).
So the biggest impact of a measure is immediately after it is introduced, because the market hasn't engineering itself to overfit to that measure.
And that is what Dan's post clearly says, that immediately after he brings up something, there is an actual change in the industry practice that happens.
> it doesn't show off creativity.
But something like Jepsen is amazingly useful because it is a creative nothing-up-my-sleeves end-run around first-party testing - actual third party testing for the product is the red-team to a QE team doing the blue-team work. The problem is that a lot of people leave that to the same vendor who is selling the product with its promises from their sales team - it is hardly an open book into the bugs open list (I work in open-source, so I have to wash my dirty laundry in public).
As a side not to this, I had a meeting in the last 2 days where someone told me that the "QE team is doing a good job, because the test have been consistently green as the release approaches".
And I didn't know exactly how to respond to that.