[fnbr]

Yeah, you’re totally right. I actually wrote a follow up essay about that:

https://finbarr.ca/llms-not-trained-enough/

I think the conversations were partly (largely?) a snapshot in time. I was talking to people in February/March, and all of this was much less thought through at the time. But you’re totally right. You want something like Llama, where you train a smaller model longer than Chinchilla would predict.