[vintermann]

I used to try to train models ages ago (that is, a couple of years), and I never found out how they got good results from top_k in the first place. My problem was that as soon as the model had generated an unlikely token, it was quickly steering towards spaces it had never seen during its training. It found itself in unmapped territory, where anything might happen. With high temperatures, it would then generate noisy garbage, with low temperature it would generate repetitive garbage. We tried to fight this with rollout strategies and backtracking, i.e. if you find yourself in a space where all tokens seem equally likely, you're probably in junk land.

Our tricks didn't work very well. But I didn't manage to keep up well enough to learn what worked.