[bastawhiz]

I can't get Copilot to generate Python that adds numbers together correctly sometimes. Getting an LLM to generate correct, working code for a language that hardly anybody writes anymore is almost assuredly going to lead to failure.

[encoderer]

yeah I agree but when you look at the slope not the y-intercept it’s getting obviously better.

one advantage the government would have is training/fine-tuning on a hundred million lines of domain specific cobol.

[bastawhiz]

The slope doesn't really matter, because the target is "better than a human, and able to identify and fix its own errors". The slope will decrease as you approach this threshold.

It's also wildly bad to plan to train and fine tune on code that you know has bugs. Already we have Copilot generating code with trivial vulnerabilities because that's what it's trained on.

[encoderer]

Roughly ~all code has bugs. We were all trained on trivial examples and buggy code - just like llms.

Honestly I think editing code will be easier than creating wholly new applications to precise spec.