Hacker News new | ask | show | jobs
by onion2k 18 days ago
The problem that I'm finding in my own work on figuring out the impact of AI is that there's just no reliable way to connect things directly to AI usage. Most of the tooling does things like "The user used AI on the same day that they opened this PR, therefore we'll assume the AI was used to write the code in the PR." In a mature AI-driven org that might be true, but in the rollout phase of an AI experiment it absolutely isn't.

Until there's a good way to fix that gap in the data any measure of AI impact is going to be horribly flawed.
2 comments
saghm 18 days ago
It's wild how we need to rediscover Goodhart's Law every time we have a new thing we think might increase productivity.
link
sorokod 18 days ago
Completely disable the usage of AI for time period t. Compare the impact on whatever before t and during t.
link
pqtyw 18 days ago
In any complex organization that might take several quarters if not more, though.
link
sorokod 18 days ago
Possibly but this is not related to AI
link
pqtyw 18 days ago
How can you compare something when you can't see or measure the outcome for an undetermined amount of time?
link