Hacker News new | ask | show | jobs
by Jensson 1231 days ago
> Code generation is moving extremely fast

Correction, it was moving fast a few years ago, it isn't moving fast right now. I can understand being a little worried a few years ago, but today? They are clearly stuck, we know the capabilities of current models and they aren't threatening anyone, if it was easy to improve then they would have made huge strides from a couple of years ago but they haven't.
1 comments
izzygonzalez 1231 days ago
That’s objectively wrong.

Codex, AlphaCode, both surpassed by CodeRL on the challenging APPS benchmark last year. Meta working on InCoder. Microsoft working on UniXCoder…

Future research directions are pretty clear from where we stand. That includes iterative methods, reinforcement learning, text diffusion, etc. No one is stuck.

link
Jensson 1231 days ago
Codex just barely surpassed it on easy questions but did worse on harder ones. AlphaCode is significantly better on harder questions, but significantly worse on easy questions. That isn't extremely fast development, they are mostly moving sideways, trying to improve one part of the metric hurts the others.

https://paperswithcode.com/sota/code-generation-on-apps

Development in these areas was very fast in the 3 years between transformer networks were invented and roughly GPT 3 was done. But in the 3 years since GPT-3 not much has changed, we see a lot of "we applied a large network to a new problem and found X" since then, but that isn't new performance, its just a new result with the same thing we had around back then.

link
steve1977 1231 days ago
unixcoder… they had to name it like that, right?
link