[Gred_papa_dance]

I need more info:

* where is data (make data) how create new my own data, (questions for chat?) * how create a tokenizer (meybe separate) * how stop the code, how many memory need, how setup size of context etc. * how creating a LORA or learn with new data. * how quantize model?

In my opinion this is great idea but making a Ruby extension will be goot way to increase users using this code.

[markusheimerl]

the data gets downloaded via curl from huggingface - sure you can make your own data, simply dump all text you want the model to be trained on into "corpus.txt" and skip "make data".

As the tokenizer adds substantial complexity, this implementation does not include any tokenziation logic and works on raw bytes. Feel free to add your own tokenzier with the help of the coding model of your choice.

You can stop the training using CTRL+C You can train on as little memory as you have. Simply reduce batch size and/or model dimensions in train.c You can change the context window size in train.c via the "seq_len" variable.

Regarding Ruby, LORA and quantization I'll have to refer you to the coding agent of your choice.

[ewew53]

Meybe add a simple step betwen start and train:

convert text data to binary data. This help converting a differend data.

(please make 8 bit format, 16, 32 bit format)