[Anamon]

As always in these discussions, I think people compare apples and oranges. LLMs are great with in-distribution solutions for solved problems with a lot of relevant prior material and in established technologies. Frontend stuff works great, for instance.

But for novel solutions, complex business logic, things deeply integrated with external systems... the code generation quickly turns terrible and useless. Especially if it's in anything but Python, Java, or JS.

Most of these differences in results end up being about differences in application. LLMs suck at out-of-distribution material, inherently.