[qudat]

I'm still thinking this through but I was arguing this position to colleagues to some shock: LLM's are a race-to-the-bottom and frontier models will not be able to afford to work on coding specific models (or coding features at all) in the very near future.

27B is already really good at coding-specific tasks. Fundamentally, there is little innovation on the core architecture: LLMs are all designed essentially the same, with minor differences in how they are trained. They are all feed-forward multi-headed attention models; it doesn't matter if it's a 4B model or a 1T model, that's just scale.

Further, the frontier models cannot afford to innovate: they have to scale as quickly as possible to "beat out" their competition. The frontier models fundamentally will not create the next "attention is all you need" monumental jump in AI.

Frontier companies are stuck on scale with zero capacity to innovate. You cannot point capitalism at "basic science research" and expect any ROI. This is a known reality. Innovation is much more indirect and a "random walk" style of knowledge acquisition.

Finally, these LLMs are quite literally designed with a human-in-the-loop, and we do not give ourselves enough credit for how well we ourselves tool-call. We are doing a lot of heavy lifting to make these models useful and you cannot simply remove us from the equation without also removing ourselves from the training pipieline.

[lugu]

There hasn't ever been in human histore more incentive to innovate than today, and you think, the best lab won't innovate. That is crazy. It is like anyone can do AI research. Of course there will be new architectures. We just discovered the steam engine and the combustion engine is coming.