[mattas]

In my experience, it’s basically impossible to accurately measure productivity of knowledge work. Whenever I see a stat associated to productivity gain/loss I get skeptical.

If you go the pure subjective route, I’ve found that people conflate “speed” or “productivity” with “ease.”

[jdlshore]

The METR study has a unique approach to measuring productivity. It’s the only one I’ve seen that holds water.

I don’t think I can do the approach justice here, but the short version is that they have the developer estimate how long a change will take, then randomly assign that task to be completed with AI or normally and measure how long it actually takes. Afterwards, they compare the differences in the ratios of estimates to actuals.

This gets around the problem of developer estimates being inaccurate by comparing ratios.