[marmada]

I think it is strongly likely that Copilot will be 1 million times better by the end of the decade. Increasing compute, new techniques, etc

The top comment on this post is complaining about a bug in Copilot generated code. 1 million times better Copilot won't do that.

[truculent]

If you use a formal system with a checker/compiler, there are certain properties that you can guarantee with essentially total certainty. I’m not sure a stochastic system like ML can ever do that.

Perhaps there is research on providing mathematical guarantees and constraints on complex models like neural networks? If not, it feels like it would be harder to give a model a high degree of control. Although embedding the model in a system that did provide guarantees (e.g. using a compiler) might be a pragmatic solution?

[flooo]

Stochastic methods may not be able to do so in isolation, but they can be used in tandem with other approaches such as in “A Deep Reinforcement Learning Approach to First-Order Logic Theorem Proving” https://ojs.aaai.org/index.php/AAAI/article/view/16780/16587

[matijash]

yep that would be really interesting! If e.g. the target generated code was some language with high type safety or a DSL whose compilation already guarantees certain properties.

[trention]

Given the fact that those models seem to improve logarithmically and that they pretty much trained codex on ~all available public code (maybe divide by 2) meaning the training data will at most scale with the growth rate of public repos in Github, I doubt that whole "1 million times better". Even the number of parameters won't improve with a factor of 1 million (it will not improve by a factor of a 1000 and may even not reach a factor of 100) by the end of the decade.