[chilmers]

It's weird to see people claiming that model capabilities are plateauing. It wasn't until late last year that we even had strong coding models. Imagine if, less than a year after the first iPhone launched, people claimed that smartphone capabilities were "plateauing" because Apple hadn't yet launched a new phone. And it seems the issue is less than "models aren't getting better" than, "models are good enough to handle 99% of the coding tasks people give to them".

[IsTom]

I wonder how's model improvement/dollar invested ratio going. If gains are made by simply spending increasingly vast amounts of cash that's not going to last.

[rimliu]

People claim what they see. I see no improvement since opus 4.6, quite the opossite.

[usef-]

That's only a few months old. Just because there's time between big releases doesn't mean progress stopped.

Fable seemed very clearly a step up in my one afternoon of usage. I gave it several bugs that other models had failed at repeatedly (in a mess of a vibe coded side project) and it fixed them each in one prompt.

[jijijijij]

> Just because there's time between big releases doesn't mean progress stopped.

No, but progress not stopping doesn't mean it's not plateauing. I believe 'plateauing' is understood as the process of approaching a plateau, not being stuck on a plateau already. So, the question is about the rate of progress, not its existence.

[usef-]

I guess we draw a different line then. This year has been full of a lot of great releases so far. Most normal people didn't even use Agents before January. It does not at all feel slower than previous years.

HN commenters have been saying that LLMs plateaued ever since the first ChatGPT release.

6 months ago:

> LLMs are amazing, but they have reached a plateau.

https://news.ycombinator.com/item?id=46109534

1 year ago:

> generative AI has languished in the same place, even in my kindest estimations, for several months, though it's really been years.

https://news.ycombinator.com/item?id=43085885

2 years ago:

> 2024 has seen nothing substantially good and the only notesworthy thing is this article finally hitting into the public consciousness that we are past of the AI peak and beyond the plateau and freefalling has already begun.

https://news.ycombinator.com/item?id=42125888

---

Many more that I haven't time to look up.

I think the present just always feels slow.

[jijijijij]

It doesn't matter where we draw the line, or what someone else said at some point. I merely pointed out your argument wasn't a sound rebuttal. Criticizing subjective experiences to assess the situation is, but that cuts both ways.

> HN commenters have been saying that LLMs plateaued ever since the first ChatGPT release.

Your earliest example was 2 years after release, when LLMs were already widely used and there is literally a source to support the claim. Now, you need to show research efforts, time and resource investments, ... are producing proportional results, disproving diminishing returns. If there are diminishing returns, LLMs are plateauing.

Also, manual capability extensions, or use/edge case adaptations, which may improve subjective usability are not exactly advances in AI as technology. LLMs still hallucinate. LLms still fundamentally struggle with certain classes of problems (e.g. counting), but you increasingly need to come up with different problem dress-ups because of targeted interventions to manufacture hype and a limited supply of test cases. Can you make the case AI got actually more intelligent, fundamentally? That is, not an increase in case specific usability, but a decrease in fundamental limitations. And is this proportional to improvement efforts?

[usef-]

> but you increasingly need to come up with different problem dress-ups because of targeted interventions to manufacture hype and a limited supply of test cases. Can you make the case AI got actually more intelligent, fundamentally? That is, not an increase in case specific usability, but a decrease in fundamental limitations. And is this proportional to improvement efforts?

I think you're setting up expectations here that nothing could falsify.

Any advance could be dismissed as "case-specific", couldn't it?

What's something concrete that you would judge as a fundamental step forward?

My own claim would be that the increase in long-horizon tasks without heavy harnesses is a sign of a decrease in "fundamental limitations".

METR's time-horizon tests are going up, and that seems to match actual problems LLMs are solving in daily usage. Eg. Agent workloads. Or games: see the newest models being able to beat longer-term tasks like Pokemon from just a minimal vision-only harness. Models last year had cheating harnesses and yet still failed.

> I merely pointed out your argument wasn't a sound rebuttal. Criticizing subjective experiences to assess the situation is, but that cuts both ways.

My original "doesn't mean progress stopped" comment was replying to the "no improvement" guy, not the plateau guy.

Yes, I agree there's vibe-based judgement happening on both sides. I was quoting old HN not to make a "boy who cried wolf" argument, but to say that feeling a vibe of slow progress is not enough, since people repeatedly have expressed that vibe.

> Your earliest example was 2 years after release, when LLMs were already widely used

Fair. I was going backwards and thought a few would suffice. There was a lot of doubt earlier too if you search. From March 2023, four months after chatgpt "We are heading towards a limit and the AIs aren't getting much better." https://news.ycombinator.com/item?id=35332537

> Now, _you_ need to show research efforts, time and resource investments, ... are producing _proportional_ results, disproving diminishing returns. If there are diminishing returns, LLMs are plateauing.

Yes, it's a good point, though we don't have good numbers on investment so it's hard to answer in either direction. But the METR speedup in 2024-25 appears to be mainly from RL/algorithm gains not training compute, judging as an outsider.

"Is progress efficiency per dollar decreasing" is a different claim to the earlier "is progress slowing?", though.

> LLMs still hallucinate. LLms still fundamentally struggle with certain classes of problems (e.g. counting),

Measured hallucination rates have gone down, and don't seem to be an issue for achieving things, as per METR and other results. I don't see hallucination issues in practice as all real work I know about includes the ability to validate.

If by "counting" you mean the "counting r's" issue, that's a tokenisation artifact (the models are blind to individual characters): ask it to count in Python and it's fine (ie. reasoning is not the problem).

(the viral online counting videos of voice-mode are using an ancient gpt4o-based model that is trained to keep answers short. I do think this is giving much of the public the vibe that current text models are worse than they actually are)

[qtk8]

Which is not even 5 months old.