Hacker News new | ask | show | jobs
by agent_turtle 323 days ago
The article you gave is derived from a poll, not a study.

As for the quote, I can’t find it in the article. Can you point me to it? I did click on one of the studies and it indicated productivity gains specifically on writing tasks. Which reminded me of this recent BBC article about a copywriter making bank fixing expensive mistakes caused by AI: https://www.bbc.com/news/articles/cyvm1dyp9v2o
1 comments
keeda 323 days ago
The quote is from the paper linked in the article: https://s3.amazonaws.com/real.stlouisfed.org/wp/2024/2024-02...

It's actually based on the results of three surveys conducted by two different parties. While surveys are subject to all kinds of biases and the gains are self-reported, their findings of 25% - 33% producitivity do match the gains shown by at least 3 other randomized studies, one of which was specifically about programming. Those studies are worth looking at as well.

link
foolswisdom 323 days ago
It's worth noting that the METR paper that found decreased productivity also found that many of the developers thought the work was being sped up.
link
keeda 322 days ago
Yes, self-reporting has biases and estimating tasks is still a fool's errand, which is why I noted that the estimates from these surveys matched the findings from other RTC studies.

However, what doesn't get discussed enough about the METR study is that there was a spike in overall idle time as they waited for the AI to finish. I haven't run the numbers so I don't know how much of the increased completion time it accounts for, but if your cognitive load drops almost to 0, it will of course feel like your work is sped up, even though calendar time has increased. I wonder if that is the more important finding of that paper.

link