Hacker News new | ask | show | jobs
by krupan 5 days ago
"Unfortunately, given participant feedback and surveys, we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools."

That's more than a caveat. That says, this study is unreliable. And why? "The primary reason is that we have observed a significant increase in developers choosing not to participate in the study because they do not wish to work without AI."

So, maybe people are fooling themselves, just like this article suggests (I almost wrote asserts, but saying, "I think..." isn't really an assertion)
3 comments
kalkin 5 days ago
The people who chose to use AI would be the ones who would be fooling themselves, right? But they measured actually positive productivity there.

METR's concern makes most sense if it's about the inability to measure a consistent trend over time, not about the validity of the study for the population that participated.

link
dcow 5 days ago
> The primary reason is that we have observed a significant increase in developers choosing not to participate in the study because they do not wish to work without AI.

Why do you think this is? Who wants to spend an afternoon gluing API calls together when the robot can do it in 5 minutes?

link
notnullorvoid 5 days ago
If you look at the data 25% of the people who reported attempting a task without AI did not complete it. Seeing as the time spent on AI tasks was not that much higher than the time spent on completed Non-AI tasks, I suspect it was less about unwillingness, and more about inability.

We should really be studying AI related cognitive and/or skill decline more heavily.

link
leecommamichael 5 days ago
Don't let the downvotes get you down. They don't know who funds or constitutes METR. If you actually read these papers, you'll find they cite themselves 4-5 times each publication. They also consider "hosting an HTTP server using Python" to be a "long time horizon" task.
link
kalkin 5 days ago
The OP cited a METR study... if you don't think they're credible, fine, but it's hard for me to see how it could be intellectually honest to cite them when their research is helpful to your argument and discredit them when it's not. One might even call this approach "fooling yourself".
link
leecommamichael 5 days ago
The OP is not citing their claims, but their commentary. Do you understand why I find that to be different than citing their findings? We're not talking percentages here.
link