[simonreiff]

I really wish this study had chosen to focus on more useful endpoints like revenue, profitability, customer retention, conversion rate, etc....at a minimum, don't look at PRs merged without also looking at bugs in production code, number of incident reports, features shipped, debt log canceled, etc. Even if you only want to consider merged PRs, you should weigh the productivity of the reviewer and whether they are also 24% boosted or else it's just an accounting trick.

[onion2k]

The problem that I'm finding in my own work on figuring out the impact of AI is that there's just no reliable way to connect things directly to AI usage. Most of the tooling does things like "The user used AI on the same day that they opened this PR, therefore we'll assume the AI was used to write the code in the PR." In a mature AI-driven org that might be true, but in the rollout phase of an AI experiment it absolutely isn't.

Until there's a good way to fix that gap in the data any measure of AI impact is going to be horribly flawed.

[saghm]

It's wild how we need to rediscover Goodhart's Law every time we have a new thing we think might increase productivity.

[sorokod]

Completely disable the usage of AI for time period t. Compare the impact on whatever before t and during t.

[pqtyw]

In any complex organization that might take several quarters if not more, though.

[sorokod]

Possibly but this is not related to AI

[pqtyw]

How can you compare something when you can't see or measure the outcome for an undetermined amount of time?

[dzonga]

negative ROI - when you start using actual business metrics.

it's in everyone's best interest to keep those numbers covered.