[jcgrillo]

Right, but most of them don't already work as software engineers, at least it hasn't been my experience of my colleagues. However, companies that are aggressively adopting AI coding tools aren't (at least nobody has shown it) getting better by any metric. So, what gives? Why, generally, isn't this kind of success story common?

[rincebrain]

Because most tasks aren't "refresh this code from 10-20 years ago"?

If you needed an old piece of code at $WORK, you probably already paid the tax of refreshing it or replacing it.

This sort of task is similar in nature to something like "I have a 25yo unmaintained Linux driver, let's refresh it for modern Linux" - a great demonstration of the efficacy of these tools if you have the right-shaped task, but not a task that comes up repeatedly in most people's days.

[jcgrillo]

That's a good point, this does seem qualitatively different. Like dialect translation. In that case the specification is really precise, it's just the old code. Building something new or adding functionality to existing software the spec is almost guaranteed to be more vague.

EDIT: this specificity seems important for language models but the harder I think about it the less sure I am that it's the right intuition..

[rincebrain]

My intuition is that what makes them well suited is that the transformations on the input desired are well-defined and frequent tasks - e.g. any other software that migrated from, say, SDL1 to SDL2, or had to move from gcc 3 to 4, or Sun cc to gcc, had to have these transformations in their source history.

IOW, "there is probably very little stopping you except time from having written Coccinelle patches to mechanically do most of these transformations".