[8note]

does lora do a good job at teaching the model new things that werent in the training data?

without trillions of examples of following instructions at a million context length, im not convinced the behaviour is in the weights to begin with

[irthomasthomas]

No. Lora is for tuning behaviour and how the model applies what it learned in training. Teaching a model new facts is still expensive.

[didgeoridoo]

There was a recent-ish paper[0] from Sakana AI about baking facts from a document corpus into a LoRA adapter. Claimed near perfect recall on very large needle-in-haystack testing. Haven’t tried it myself though.

[0]: https://sakana.ai/doc-to-lora/

[irthomasthomas]

interesting... thanks.