[_bobm]

But, guys, when you say Claude/ GPT models, do you stop to think what are these "models"?

One day I thought about how can GPT send thinking parts one after another with a markdown header summary of the thinking block itself. Just think about it.

As a matter of fact, think about these operations, api endpoints, observe their output.

These so called SOTA models are not what meets the eye, and are not at all comparable in the infra department to local models. There is crazy orchestration going on due to the scale of these operations. But also these hard constraints lead to innovation. Innovation nobody speaks about.

I wouldn't say we cannot catchup, but serving our local models through llama, vllm is just the A, B, C of it all. In reality I think what is needed is a replication of said orchestration which I hinted at above.

The SOTA models are a deep orchestration of multiple models operating together it isn't a single model. As such no single model ever will catchup to them until it replicates through training first and then maybe through model architecture this orchestration.

Finally, I would wager that the SOTA "models", as one of these models in this orchestration setup, as served for general consumption, are not so much more capable than qwen 3.6.

I am sure that if you change your perspective you will start noticing the scale of the "magic".

[XCSme]

> The SOTA models are a deep orchestration of multiple models operating together it isn't a single mode

I don't understand, why does it make you think this is the case?

> how can GPT send thinking parts one after another with a markdown header summary of the thinking block itself

Can you give an example?

[_bobm]

> Can you give an example?

Sure, connect opencode to an openai/chatgpt endpoint and use it. You will notice multiple "thinking" parts per "turn".

I put all of these in quotation because... they are part of the orchestration game. For example, it is not known if the thinking parts of a particular turn are chain of thought thinking summaries or just plain response which is masquaraded and thus orchestrated into appearing as thinking.

Further notice the cadence, word choice and sentence formation. Notice sentence construction. Notice "thinking part" construction and sequencing.

There is pretty heavy orchestration.

> I don't understand, why does it make you think this is the case?

Because not all tokens are equal. And if you waste expensive tokens on mundane tasks you will go out of business. This is the reason.

As I said, if you observe the output from these api endpoints you will notice it.

[XCSme]

> You will notice multiple "thinking" parts per "turn"

I thought that was the code harness simply minifying the outputs. Many models now no longer return the entire chain-of-thought (to avoid distillation attacks). So yes, we don't get the raw LLM output, but I think it's just the thinking summarized, not a complex orchestration or different models.

I do agree though that now cloud models are kind of a black box, that's not only obfuscated but also changes over time. Companies seem to be changing model capabilities without notifying users, or even hiddenly serving completely different models. This is even worse via OpenRouter, with providers serving open-source models, some of them serve heavily quantized versions or even completely different models.

[_bobm]

idk what is "minifying outputs" in the context of what we are talking about. Opencode is opensource, you can find out what it is doing.

Last time I checked, OpenAI even send (in the response) the summary of the thinking part alreafy in markdown, so opencode has to remove the formatting to format it to their liking.

> Many models now no longer return the entire chain-of-thought (to avoid distillation attacks).

This is what they say: to avoid distillation attacks. And to some large extent this is true. I am saying there is a side- effect and this side- effect (depending on how tin-foilly you want to go) may be either a nice thing to have or it may be the "main reason" for all of this.

The side effect is splicing the inference, brokering requests, and what not, which brings huge benefits at scale.

This was my original point: openweights model to a sota model may be apples to oranges. So when will a local model catchup with its single cot run which is not even shaped properly: well never.

It is apples to oranges.

[XCSme]

So, are you saying that local models are maybe better than we give them credit? Because with some extra orchestration/processing we could improve the results?

[_bobm]

Yes, local models have already all that is needed, they have all the prerequisites.

But what they do not have is the correct shape, the correct approach. This is missing and it shows on multiple scales: it shows in the COT, it shows in the output itself, it shows in the infra to serve the models, it shows in the model orchestration.

This is what anthropic said one year ago:

> Finally, we've introduced thinking summaries for Claude 4 models that use a smaller model to condense lengthy thought processes. This summarization is only needed about 5% of the time—most thought processes are short enough to display in full. Users requiring raw chains of thought for advanced prompt engineering can contact sales about our new Developer Mode to retain full access.

[JSR_FDED]

This all sounds very mysterious

[_bobm]

Yes, but it isn't.