[torginus]

I just have small thing to add to this article - it mentions how the code contributed per engineer has increased as per Claude Mythos to 8x of baseline.

Now, I have encountered many times, when I asked AI to implement a function for me for which I was 100% sure a good implementation already existed in the form of an npm package, it had the tendency to go ahead and implement it on its own. Now, I usually trust battle tested implementations to be more robust, but if the AI does this (which I think is not an unique observation), you can easily balloon per engineer line generation (as can you with reduced oversight), so as always, these high level benchmarks are to be taken with a grain of salt.

[jcfrei]

Maybe Im nitpicking here but LLMs are quite literal. So when you tell it to "implement a function for me" it will necessarily write the whole thing. Changing the prompt to "find an existing implementation for this" would be more apt.

[torginus]

It certainly was a bias in this case - I asked for a feature, mentioning to give me popular libraries in this space, then it started coding up a bespoke solution that included among other things, a custom HTML parser. Then asked to really consider using a library for the whole thing and it 'compromised' by pulling in a HTML parser, but still keeping its own implementation.

This is just a singular example, but I'be noticed a strong and beyond reasonable bias for this from multiple LLMs (like not using the already included dependency)