[cubefox]

Both increasing the amount of parameters and the amount of training tokens improves results (more precisely: lowers training loss), and costs computing power. For optimally improving loss per training computing power, model size and training tokens should be increased equally. That's the Chinchilla scaling law. (Though low loss is not always the same as good results, the data quality also matters.)

Further reading: https://dynomight.net/scaling/

[snovv_crash]

An interesting corollary of this is that if you want to reduce the model size you can compensate by training for longer to achieve the same accuracy. Depending on your training:inference ratio this may be more optimal globally to reduce your total compute costs or even just reduce your frontend latency.

[cubefox]

Yeah, though I have not seen a formula which takes the number of expected inference runs into account for calculating the optimal data/parameter balance.