[qq66]

I do have a private benchmark I use for that. I have a repository with an in-progress codebase and a very large planning document, I tell the new models to simply finish everything in the planning document, and I never push those changes. All of the recent models have essentially gotten it perfect which means that the models are all equivalently good for my level of needs.

[londons_explore]

I have noticed that codebases I have used an AI on seem to perform better in future AI models than ones I have not.

This makes me think AI companies are using chat history to train the next model.

[tln]

AI generated code is usually full of comments, human code is usually less commented.

It may be that AI performs better on such code.