[eternalban]

Great, we can get authoritative answers. (I'm trying to understand the ML space and have mostly done readings, not an expert.)

I am assuming you can have n LoRA fine-tunings, say each specializing in one aspect of a coherent task, with n summers, running in parallel, and then combine them at the end? Or more generally, does LoRA enable a sort of modularizing around a core (un-merged) model?

And curious if you ever tried merging 2 or more fine-tunings and then testing the resultant single model (merge all) against the original tests to check retention?

[edwardjhu]

This paper tries something like that

https://arxiv.org/pdf/2202.13914.pdf

The gain isn't that significant. We don't understand what these low-rank updates represent, and they might not correspond to "skills" that humans have.