[hirako2000]

Is there any source you could reference. Really interested.

It would not surprise me, why would they build from scratch, every LLM is a "fork" of gpt. Did they not come up with the mixture of expert idea though ?

[bpavuk]

and every LLM is a "fork" of Google's Transformers architecture.

everything is a "fork", if you give it a serious thought.