Hacker News new | ask | show | jobs
by realusername 41 days ago
There's also a lot of benchmark trickery going on, it's becoming harder to see how the latest models really improved.

The top models also seem to have inconsistent performance depending on the time of day and how far we are from the next release.
3 comments
bonesss 40 days ago
I’m an LLM fan, but from an engineering perspective the idea of building atop services that palpably fluctuate in capacity, performance, and capability is nutty.

Even with minor automation I feel like I can watch OpenAI and Anthropic engineers fiddling in real-time. Tuesdays behaviour changes by Thursday, 10AMs production isn’t possible at 11:30AM. Nutty.

link
targafarian 40 days ago
I chilled significantly on using Google for anything to do with business due to API (and offering) stability. (Still use Google for personal things.) But AI models seem orders of magnitude more fluid, so to my risk-averse eye, they're nothing I'd base my own business on.
link
senordevnyc 40 days ago
Imagine having a business where you're at the mercy of the fluctuations in capacity, performance, and capability that your human employees display!
link
bonesss 39 days ago
… all of which can be controlled through scheduling, and has been baked into all business operations since before money was invented.

Meatspace fluctuations are predictable, quantifiable, and manageable through planning. The changes to hosted model behaviour are not.

Literally, the prompt terms used to manage document reading can change multiple times within a day. MS and Oracle provide decades-long consistent interfaces in their data products.

link
senordevnyc 37 days ago
True, but we'll get there with AI models as well. To your point, we've had millennia to learn how to build large business operations on top of fickle humans, and we'll figure it out for AI too. Just getting started.
link
intothemild 40 days ago
Since I started running my own inference server, I've had zero degradation that I didn't do myself. Basically the only time I see it get worse is if I drop one of the quants.

Which is what I suspect the providers are doing to fit more inference on the same amount of hardware over time.

link
Barbing 40 days ago
Interesting, Claude might be doing better since I last checked:

https://marginlab.ai/trackers/claude-code-historical-perform...

There were at least a couple of these degradation trackers.

link