[daveguy]

> Often yes. In this case, it's more like they get upset when someone says something factually wrong, and then defensively changes the goalposts.

Oh give me a break. Show me one example of 1) any knob twisting that makes the underlying model better. or 2) any example of the AI providers twisting those knobs to do anything other than degrade performance for their own bottom line or safety.

The current post says: "it would be expected for a better model to use different amounts of brevity if it gets better at determining the appropriate amount."

When no, the model cannot "get better". It doesn't determine any appropriateness of response realtime except for the weights baked into it from the beginning and whatever context it can muster. If you cram enough guidance that it doesn't decide to ignore maybe you can make it more brief. But it (the model) can do none of those things.

LLM models are literally stupid by design.

[dahart]

Why did you drop the first half of the sentence in your quote? The qualification there is important context for the part you did quote. And why are you talking about “better” within a model, when the sentence you quoted was talking about 5.6 vs 5.5? The post you’re referring to did not suggest a single model could “get better”. You’ve made some incorrect assumptions.

Your comments are conflating multiple kinds of “smart” and “better”. You’re right that if all the inputs are exactly the same, it takes a new model to improve (ignoring non-determinism). But the knobs and context and harness change the inputs, and they do improve output, contrary to your claim. You’re failing to capture the distinction between what the model itself does and how the harness can boost the model’s performance. It is legitimately valid and fair to call improved performance “better”, no matter where it comes from.

This all gives me the feeling you might not have experience with or understand what’s happening in today’s harness development, and the degree to which it may be as important as the weights. There are in fact a lot of things you can do to improve a model’s performance on tasks & benchmarks, without changing the model weights. @coldtea mentioned a bunch, but the harness feedback loop, internal prompts, system prompts, skills, and requests for a model to try harder, and verify and validate it’s output all lead to improved performance, all without retraining.

I agree LLMs are stupid; they’re statistical token predictors. But somehow statistical token prediction is amazing and works much better than we imagined. The talking points about LLMs being stupid token predictors are fading now because they lack explanatory power for how good the models have become. The big surprise here isn’t about LLMs. It’s about language, and how much “thinking” and intelligence is contained in language. We don’t have a good grasp on where the line is between language and intelligence. LLMs have crushed the Turing Test into dust, and yet we don’t consider them intelligent. They often appear to understand what you ask thoroughly, can re-state it in different words, they can correct your misunderstandings or add nuance you didn’t see. All this because that’s what humans do and LLMs talk like humans.

[daveguy]

> Why did you drop the first half of the sentence in your quote?

Because this entire discussion is about the release of a new model, and models are fixed. Sure you can try to modify all the scaffolding around it, but the model is the model. It doesn't matter what you're trying to improve. You can only improve the peripheral aides. And the peripheral aides can't fundamentally fix the problems with llm models when they can't learn new relationships or facts.

You will always have to wait for a new model (like this one we are talking about) for improvements to the model.

[dahart]

> Because this entire discussion is about the release of a new model

Right. The sentence you quoted was about brevity improving with a new model. It did not suggest the model itself improving.

I’m confused why you’re stuck on this tangent. And confused why you are repeating the talking points about the model being fixed. The model is fixed - that’s true, I already agreed with you. But you don’t seem to be listening to anything else.

> It doesn’t matter what you’re trying to improve.

What do you mean? If we’re trying to improve LLM output, there are multiple ways to achieve it. A new model is one of them. Changing the inputs is another.

> You will always have to wait for a new model (like this one we are talking about) for improvements to the model.

This is true! Nobody here is disagreeing with that. The part that it seems you’ve argued incorrectly is the apparent claim that output can’t get better. Output can “improve” without improving the model.

[blackqueeriroh]

> If you cram enough guidance that it doesn't decide to ignore maybe you can make it more brief.

You are now anthropomorphizing the model yourself.

[coldtea]

>Oh give me a break. Show me one example of 1) any knob twisting that makes the underlying model better.

I mentioned several.

You're now once again changing goalpoasts to say you meant the underlying model, not the overall llm performance, even though you explicitly wrote: "Their performance depends solely on the model training before release and how well you curate the context you feed it".

So, the context curation was relevant (meaning you didn't constrain your claim to the underlying model), but now somehow all the additional tunables aren't relevant (because suddenly you're just talking about the model).

End of discussion.

[daveguy]

None of what you mentioned changes the model. Because it's a fixed model. The weights are constant. It does not learn. It only knows what gets repeatedly fed to it and those fixed relationships represented by the weights. You can pretend like that's not true, but unfortunately for VCs it is true.

End of discussion.

[coldtea]

"Their performance depends solely on the model training before release and how well you curate the context you feed it".

Wrong. The face-saving backtracking doesn't change that.

[daveguy]

The models do not get better until a new one is released. And we are already at diminishing returns. So sorry. Also sorry you don't know the difference between a model and a context, harness, router, or cache.