[steveklabnik]

From the paper:

> We do not provide evidence that:

> AI systems do not currently speed up many or most software developers

> We do not claim that our developers or repositories represent a majority or plurality of software development work

[almog]

Not sure why you quoted that part, it just says that there is no assumption for the results to be extrapolated to any codebase or any developer, setting the boundaries of the study objectives.

[steveklabnik]

You have made the claim that it does extrapolate. Which they themselves do not make.

[almog]

How is saying that it's not all clear LLM increase experienced developers productivity an extrapolation?

The only extrapolations I've seen on this thread are people shrugging it as using 6 months old LLMs so this whole paper must be invalid today.

[steveklabnik]

Okay, so, I re-read your original post:

> it's not all that LLMs increase experienced developers productivity (even when they believe that it does):

In the present, I am struggling to parse this. When I made my original comment, I understood you to be saying that LLMs do not increase productivity. Synthesizing what you're saying now with that, if I had read

> it's not at all clear that LLMs increase

then I would have understood you correctly. That's my bad!

> The only extrapolations I've seen on this thread are people shrugging it as using 6 months old LLMs so this whole paper must be invalid today.

I feel for both sides on this one. I do think that, for me personally, the models they used weren't good, but the ones that exist now are. So I do think there's some issue there. However, I don't think that makes the study invalid, if anything, it's a great way to test this hypothesis: if they do the same thing again, but with newer models, that would lend some weight to that idea. So I also think saying that this is completely irrelevant is missing the point.