[IdiocyInAction]

> Fine tuning a vanilla language model on a giant corpus of code feels like a dead end for the field, long-term. It seems obvious to me that humans are doing something more than just statistical pattern recognition and generation when we write and reason about code.

Yeah, this is the main reason why I would be interested in more examples. But, if this thing was trained on all of GitHub, I could imagine that it come up with decent-looking code for a lot of examples; a beefy, smarter Google with some rudimentary contextual understanding, if you will. Still, the presence of any mistakes is a no-go and I'd be really interested how it reacts to more realistic, specific requirements.

But yeah, I'd figure a model for code generation would have to have some kind of knowledge of syntax and semantics, rather than doing pure statistical pattern matching, to be of any real use. It would not only have to generate, but also to debug its code (I wonder whether you could do that purely with statistical pattern recognition). I might be wrong, of course, but I would be surprised if that is enough to write complex code.

[MauranKilom]

Five years ago we were already here: https://karpathy.github.io/2015/05/21/rnn-effectiveness/

Calling the field "statistical pattern matching" might be underselling it a bit, even if technically accurate on some level. I mean, syntax/semantics are clearly not the problem, those are the easiest to learn (see the paper above). If anything, I'm scared of it writing syntactically correct nonsense (or even worse, subtly-flawed-but-correct-looking code).