[oakinnagbe]

Nice implementation. Have you thought about supporting LoRA fine-tuning on top of this, or is the design too low-level for that kind of extension?

[markusheimerl]

Sure it could be extended to support LoRA finetuning but this implementation has the goal to be as lean and efficient as possible for a pre-training stack as you can be.