[in-silico]

> Keep in mind that all LLMs are trained first on text and then fine-tuned on code.

No, they are trained on a mixture of text and code from the start.