Yes i agree, but somehow knowledge got put into the LLMs head during train time, but why does it not work during inference time? Too little examples? Do llms know stuff with exactly one occurence in the training data?
What gives LLMs the ability to reason so well about math specifically is the math-specific RL training they are given where they are rewarded for chaining together reasoning steps etc in a way that results in success (e.g. a known correct result).
The difference with something completely new at inference time, that was not in the training data, or not used by RL training, is that while it will be able to use it to some extent, it has not been taught via RL how to best use in a reasoning chain.
The problem is that LLMs don't really have the generic ability to reason, so they instead need to fake it by either:
1) Fine tuning on reasoning data (very specific)
2) RL training (more generalizable, especially for math/coding)
3) Prompting that encourages "keep on going" "tree of thoughts" exploration where they may discover a reasoning chain largely by luck
Demis Hassabis has talked about combining LLMs with search (cf systems like AlphaGo) which sounds like it might be useful for math research.
RL involves weight updates, but the model is frozen after training - no inference time weight update! An RL-trained model does tend to have generic "reward maximizing" long-term goal behavior at inference time, but it's ability to correctly/fruitfully chain together reasoning steps is much dependent on RL training.
> also i think that it is reasoning exactly like us humans do. Indistinguishable
Yes, it is copying human reasoning so it will appear the same, but the difference is when you don't know what to do/try next - when you are trying to solve a problem that you have never solved before and don't know from experience what to try next. This is when having real/generic ability to reason, not just "reason from memory" matters. This is when things like human curiosity are useful : "I wonder what happens if I try this ..."
It's like that all that knowledge exists as a vague memory. I've found if I ask them specifics about any book I've read (w/o allowing tool calling), when they get it wrong their hallucinating things that are often correct or wrong in subtle ways. Are they're just completely out of pocket.
The difference with something completely new at inference time, that was not in the training data, or not used by RL training, is that while it will be able to use it to some extent, it has not been taught via RL how to best use in a reasoning chain.
The problem is that LLMs don't really have the generic ability to reason, so they instead need to fake it by either:
1) Fine tuning on reasoning data (very specific)
2) RL training (more generalizable, especially for math/coding)
3) Prompting that encourages "keep on going" "tree of thoughts" exploration where they may discover a reasoning chain largely by luck
Demis Hassabis has talked about combining LLMs with search (cf systems like AlphaGo) which sounds like it might be useful for math research.