[elpakal]

How about starting with mean time to merge a code change? I get that there are other variables that contribute to productivity but things like satisfaction, collaboration etc are extremely difficult to measure well and just IMHO pretty tangential (disclaimer - I work on a dev prod team and my entirety of last year was spent building engineering metric dashboards and discussing what to measure, and it's not easy so I get it).

[ravenstine]

While we're at it, let's measure productivity by lines of code added. /s

Using merge time is a terrible metric, perhaps worse so than deadlines because it can be more effectively weaponized. What would the incentive be for the developers other than to rush the code review process? Merges are not where you want to be rushing anything, but rather the opposite. If project deadlines are necessary, allowing code review its due time affords developers the ability to informally schedule things without sacrificing craftsmanship for what in reality is a vanity metric. Some code needs the be carefully considered and given time while other code doesn't necessarily need much review or worry at all, but no one can tell that by looking at mean time. If a developer is asked why some tasks had a longer than average mean time, then now they have to waste even more time by explaining themselves. In the worst case, the incentive to rush the review process results in more time wasted on bugs that could have been caught before they even had a chance to be merged.

Am I misunderstanding your view of how mean time to merge would be used?

[sixstringtheory]

Sounds good to me. Then I imagine the question is all about the kinds of changes that are being merged, which leads to: How many features can be produced per unit time? How many bugs are produced per feature that must be fixed therefore slowing the rate of feature production?

The questions of whether features are appreciated by users, or which bugs should be fixed or not, or if a product is feature complete or needs more, are questions of business, and not developer, productivity and efficiency.

And regarding documentation, I consider that an integral part of code/software that can be judged similarly w.r.t. quality and impact, having its own features and bugs.

[blurker]

Using a metric like mean time to code change could incentivize brute forcing work. Developers who are aware of the metric may work unsustainable hours and that could lead to burnout. Also, they may try to cut corners on tests, reviews or documentation in order to ship more things faster.

I think that qualitative metrics like satisfaction and collaboration could be helpful, especially when combined with traditional metrics like mean time to merge a code change. Taking my previous example of overworking or cutting corners to achieve high numbers, a qualitative metric for something like satisfaction might indicate a problem where a work output metric wouldn't.

But I think that any combination of metrics will be an oversimplification that could lead to problems if they are the only thing that matters. I'm not sure where the balance lies. I like that metrics can offer an objective view of performance and make it easy to spot trends. But I am wary of them oversimplifying things and dehumanising the team.

[elpakal]

> Using a metric like mean time to code change could incentivize brute forcing work. Developers who are aware of the metric may work unsustainable hours and that could lead to burnout. Also, they may try to cut corners on tests, reviews or documentation in order to ship more things faster.

I think this depends on how strongly leadership/management tries to use this metric to change behavior, but in a company like mine where we have a lot of business dependencies, contracts etc which rely on predictions of throughput, this is an important metric for us to predict timelines. It's not used against a team or platform and it is primarily used internally by management and not so much engineering teams which I think is the right way to use it.

[sixstringtheory]

> It's not used against a team or platform and it is primarily used internally by management and not so much engineering teams which I think is the right way to use it.

Absolutely. It’s the old descriptive vs normative saw. We should be very interested in measuring how productive we are. But how do we really know how productive we should be?

I think trying to answer that is hard to impossible for a developer, because you can always automate or abstract further, but the cost to do so marginally increases and you probably won’t know the full cost until it’s already realized, at which point, requirement and prediction are obviated.

I know I’ve had my fair share of negative experiences with a scrum team that wanted ever faster velocity and used burndown charts as a stick with no carrot in sight.

[blurker]

That makes sense to me. Then in your situation, assuming you need those metrics and they are helpful, it may be helpful to add a metric like "satisfaction" or something that could counterbalance the negative effects that the other metrics could have.

I could see how a satisfaction metric would counterbalance a work output metric. The team will probably feel less satisfied if they know that corners are cut or if they are having to work unsustainably.

[sdesol]

> How about starting with mean time to merge a code change?

I'm currently experimenting with using "Mode time" as I think it is less susceptible to data skew from outliers. See example below:

https://oss.gitsense.com/insights/github?p=days-open&q=days-...

For popular open-source projects that I used in the link above, the Mode time to merge is less than a day which is quite good in my opinion. And as you can expect, if you look at larger pull requests (>=10 file changes) the overall percentage drops by half as the link below shows:

https://oss.gitsense.com/insights/github?p=days-open&q=pull-...

I think what the link above shows is, you can't just willy-nilly use merge time to measure productivity, since there are a multiple variables at play.

Full disclosure: The link that I referenced is my tool