[chowells]

Well, we absolutely have models in our heads. That's how we can understand what programs do.

It's possible our minds are also huge and complicated neutral networks, though I suspect that description is incomplete at best.

But the point is that current tools are trained on text generation. Something that would change programming would have to train on the meaning of programs. It's a rather different task, as it's no longer statistical. Doing it properly requires metacognition as well, to avoid falling into the trivial inconsistencies in most programming languages. And connecting that with real-world tasks that it hasn't seen before would require an understanding of the real world.

I'd call something with all of those capabilities AGI. I honestly don't think any system short of that will ever be more than an autocomplete, because it can only ever fit things together based on some statistics.