[vessenes]

Yeah, good idea! I'm not sure if you would be successful mixing LoRA + functional tokens. If you could, that would be great. Then You could ship very light LoRA packs with repositories.

Their LoRA training was I think against their finetuned model, not Gemma-2B directly. But, seems worth playing with -- could be super useful.

[AlexChen-NexaAI]

Yes, you can do that! We have experimented about this idea. The lora training is reported in the paper already, and it works well. I think it would be very interesting.