[HarHarVeryFunny]

What gives LLMs the ability to reason so well about math specifically is the math-specific RL training they are given where they are rewarded for chaining together reasoning steps etc in a way that results in success (e.g. a known correct result).

The difference with something completely new at inference time, that was not in the training data, or not used by RL training, is that while it will be able to use it to some extent, it has not been taught via RL how to best use in a reasoning chain.

The problem is that LLMs don't really have the generic ability to reason, so they instead need to fake it by either:

1) Fine tuning on reasoning data (very specific)

2) RL training (more generalizable, especially for math/coding)

3) Prompting that encourages "keep on going" "tree of thoughts" exploration where they may discover a reasoning chain largely by luck

Demis Hassabis has talked about combining LLMs with search (cf systems like AlphaGo) which sounds like it might be useful for math research.

[fred123123]

ok but why no inference time RL? also i think that it is reasoning exactly like us humans do. Indistinguishable

[HarHarVeryFunny]

RL involves weight updates, but the model is frozen after training - no inference time weight update! An RL-trained model does tend to have generic "reward maximizing" long-term goal behavior at inference time, but it's ability to correctly/fruitfully chain together reasoning steps is much dependent on RL training.

> also i think that it is reasoning exactly like us humans do. Indistinguishable

Yes, it is copying human reasoning so it will appear the same, but the difference is when you don't know what to do/try next - when you are trying to solve a problem that you have never solved before and don't know from experience what to try next. This is when having real/generic ability to reason, not just "reason from memory" matters. This is when things like human curiosity are useful : "I wonder what happens if I try this ..."