[mikeagb]

The question of how to fine-tune to teach LLMs facts/knowledge is definitely something we're interested in exploring more in future work. The common opinion seems to me at least to be that fine-tuning is more to teach the model how to use the knowledge it already has to complete a specific task rather than to instill new knowledge, and that RAG should be sued to provide more specific context. However, I personally believe there is potential in fine-tuning for "memorization" or learning, and am excited to see new developments in the field.