|
|
|
|
|
|
by CodesInChaos
14 days ago
|
|
|
> LLMs seem to get tragically stuck on certain patterns. That is likely an artifact of the fine-tuning process: > Once a style tic is rewarded, later training can spread or reinforce it elsewhere, especially if those outputs are reused in supervised fine-tuning or preference data. > That creates a feedback loop: > * Some rewarded examples contain a distinctive lexical tic. > * The tic appears more often in rollouts. > * Model-generated rollouts are used for supervised fine-tuning (SFT). > * The model gets even more comfortable producing the tic. https://openai.com/index/where-the-goblins-came-from/ |
|
|