But LLMs don’t seem particularly good at inventing new ways to code (or write, or…). It’s literally all derivative. So what happens in 10 years? Are we headed for a great stagnation?
> But LLMs don’t seem particularly good at inventing new ways to code (or write, or…). It’s literally all derivative.
I think the key part is how much thought goes into something.
Optimistically, LLMs are good at taking unstructured input, and (probably) producing the intended output from that. -- This allows for an interesting new way of coding: a set of instructions don't need to be as rigorous as a shell script, but can be natural language.
That part surely extends creativity. An LLM will be familiar with domain ideas I'm not, even if an LLM is completely disinterested in doing things.
Pessimistically, I think it's still not clear what the right way of interacting online with all of this is (other than clear expectations of "no AI")... in some sense LLM output is worthless to share, in the sense that I'm just as capable of asking the LLM to output something as anyone else is.
Looking at LLMs applications to math might be instructive. A year ago when they had some preliminary claims/results, people would hypothesize they had the answer implicit in their training data (and so were being "better search", but fundamentally doing derivative work). This may have been substantiated sometimes, I forget.
Recently the tune has changed somewhat, say with LLM's approaches to Erdos problems (and in particular the unit distance problem. The LLM solution here spurred progress on another large problem, namely https://arxiv.org/abs/2605.28781 ). There have been no claims that the LLMs work on the unit distance problem was derivative, and I've seen mathematicians claim it would have been accepted to a top journal (say Annals).
In spite of this, the capabilities of LLMs within mathematics are still limited. LLMs seem decent at
1. "constructions", e.g. where you claim \exists object with certain properties. It can help if the verification that the object has these properties is efficiently computable, but I don't believe this sort of verification was used for the unit distance problem.
There are other areas of math that LLMs so far are less adapted to, for example
2. impossibility results, or showing \lnot \exists object with certain properties, or
3. "abstraction building". Often in math results become much easier to obtain if you have "the right definition". Grothendiek was famous for this, as is e.g. Scholze currently.
These claims are based off of current public results via LLMs. It's possible capabilities will develop further. But also, in hindsight, it is natural that LLMs would be better at the thing they ended up being good at.
I'm unsure if there is a way of extracting from this insights to programming/writing. Plausibly, you could see LLM's developments of PoC exploits as similar to (1) but for computer science. It is a concrete "construction" that is efficiently verifiable. (2) would suggest trivial observations that it would be hard for ah LLM to show that a program does not have vulnerabilities. I'm not sure if there are less trivial observations. Finally, (3) might be what you're bemoaning. In simple language, it would currently be surprising if LLMs could create useful, novel, design patterns/abstractions.
I think the key part is how much thought goes into something.
Optimistically, LLMs are good at taking unstructured input, and (probably) producing the intended output from that. -- This allows for an interesting new way of coding: a set of instructions don't need to be as rigorous as a shell script, but can be natural language.
That part surely extends creativity. An LLM will be familiar with domain ideas I'm not, even if an LLM is completely disinterested in doing things.
Pessimistically, I think it's still not clear what the right way of interacting online with all of this is (other than clear expectations of "no AI")... in some sense LLM output is worthless to share, in the sense that I'm just as capable of asking the LLM to output something as anyone else is.