I have experience working with Large Language Models (LLMs) during my tenure at my previous company. However, one aspect I didn't get the opportunity to explore there was fine-tuning my own LLM, so I decided to experiment.
While there are many pieces of documentation available on this topic scattered across the Internet, putting them all together is not obvious. Therefore, in this repository, I present the full training code, from scrapping, to generating the dataset to the finetuning itself.
The whole thing takes a few days to complete, and it does work on my AMD GPU!
While there are many pieces of documentation available on this topic scattered across the Internet, putting them all together is not obvious. Therefore, in this repository, I present the full training code, from scrapping, to generating the dataset to the finetuning itself.
The whole thing takes a few days to complete, and it does work on my AMD GPU!