[riku_iki]

I didn't get one detail: they selected 6B transformer as baseline and compared it to 7B Griffin

Why wouldn't select equal size models?..

[szundi]

They probably had them for some reason and it was cheaper not to retrain one of them again

[riku_iki]

Its just performance comparison is misleading then, they report marginal improvements which is expected just because of models size differences..

[GaggiX]

It also performs better on any other size.

[riku_iki]

They have baseline transformer of max size 6B in tables. Other models are trained on very different data and probably differently.

[GaggiX]

All the MQA transformers, Hawk and Griffin are trained on the same MassiveText dataset so no.

[riku_iki]

Yes, but MQA is limited to 6B size, while "other" larger non-RNN models in table(Llama-2) are not trained on the same dataset, and Hawk and Griffin are 7B. Sorry, I don't understand your point.