|
|
|
|
|
|
by mufasachan
980 days ago
|
|
|
Disclaimer: This is just my intuition, I do not have knowledge about LoRA on small models. It's possible that does not work. LoRA (for Low Rank) benefits from the "small changes" introduced during finetuning of a model. The update of the weights has a low rank. If you take a smaller model, it might induce that the rank is not so low, resulting in degradation in metrics by LoRA compression. I would be interested to see if LoRA still has a benefit in this configuration. |
|
|
Pedantic, but it is actually for Low Rank Adaptation.