Y
Hacker News
new
|
ask
|
show
|
jobs
by
CuriouslyC
2 days ago
The pretraining data isn't getting muddled, the instruction just gets in the lossy middle, and there's less for attention to grab on to. If you repeated the prompt with a hook every turn, it'd stay quite sharp.