[theLiminator]

Seems like the way to go for any smaller models is to only use the low reasoning levels, and for anything where you'd want it to reason harder, to just use a larger model.

In effect, high reasoning only makes sense when you're using the frontier model and need extra performance (higher levels of reasoning are never pareto optimal unless you're at the largest model size).

[adam_arthur]

I've found disabling reasoning entirely but adding a "reason" to the JSON response from the LLM to work significantly faster and consume many fewer tokens for narrowly scoped prompts.

At least for Claude family models.

e.g. {

  "reason": "<Describe why you picked this result>",

  "selection": "<The number of the value you selected>"

}

I'm sure native reasoning produces more accurate results, but for my use case the quality was about the same, and the model would reason for thousands of tokens in native reasoning vs just 1-200 with response level reasoning.

Again, to be clear, this is for deterministic/pipeline style workflows, not agentic/coding use.

[grim_io]

What you are doing, is producing an unnecessary summary of the result, not reasoning that models do to come up with the result.

I don't get what value you get out of this.

[adam_arthur]

This is the same concept as Chain of Thought.

Just that when "native" thinking is used it's hidden from the end result via special tags. If you force a model to reason about a result before producing the result, you get more accurate results.

Because "reason" comes before the selection, it has to think through why it is producing the result beforehand (e.g. produce a block of text that makes the correct answer statistically more likely to be sampled from the distribution. Giving it a property name does influence the direction of the thinking, but it's the same concept. You can call the property "yourThinking" too.

It sounds like you may be thinking too highly/mysteriously about how LLMs work.

At the end of the day they stream completely unstructured text outputs and all behaviors on top of that are just parsing XML-like tags to do tool calls, hide thinking etc (which they were trained to produce in certain circumstances).

There is no special "thinking process" it is a stream of text in the response that is simply wrapped in <Thinking> tags (or similar)

[grim_io]

I'm aware how reasoning works. But maybe I am misunderstanding how you use it.

Because from what I see, you are not getting a chain of thought, but just a summary of a non-chain-of-thought answer, with no reasoning either explicit or implicit.

[adam_arthur]

If the LLM has to write out the reason, it is "thinking".

Whether you ask it to give <reason> or <thinking>, both will produce similar chain of thought processes.

To explain the "reason" requires producing thoughts that justify the answer that is not produced yet, because it writes the JSON linearly from top to bottom.

Not sure where the misunderstanding is coming from.

[docheinestages]

My experience with using low reasoning effort has been nothing but a waste of time. Claude often keeps guessing, not calling tools to ground itself, and basically at the end I end up wasting the same amount of tokens or just switch to Opus on xhigh. It's been a terrible experience.

[mwigdahl]

Not to sound like an LLM, but that seems exactly right to me. Use it as a cheaper, high-functioning task subagent and lower reasoning for a master Opus session. As long as not every portion of your task requires maximum intelligence, you should come out ahead.

[user43928]

Won't any input be charged uncached, and the output of the small model charged again as uncached input to the bigger model?

I don't know whether that comes out ahead compared to just staying with the better model in the first place.

[mwigdahl]

It's a good question, but for multiturn conversations even cached context adds up quickly. My experience has been that spawning off subagents for defined tasks in a large overall plan generally makes me come out ahead.

I'm sure folks' mileage will vary though.

[noisy_boy]

I asked this question and was told that even if it is counter intuitive, medium will be more cost efficient due to caching. Changed to medium, blew my budget and went back to low.