[sosuke]

I’m not well read on LLMs in spite of using them daily. The increase in performance seems incremental. As if they added another 0 to the number it might only go up the same percentage in output.

So I assume that the number is just one facet of increasing output quality. Is that a safe assumption? Like throwing more energy at a problem to improve output it only goes so far.

[gostsamo]

You can improve results with cleaner datasets and if you prioritise a certain goal like conversation, code compleation or reasoning.

[andai]

I'm reading the Textbooks Are All You Need paper, which goes into this idea. The result of that research was Phi 1, and eventually Phi 3 (released a few days ago).

[tosh]

There has been incremental progress for about 1 year from open weight models worse than GPT-3.5 to models in the area of GPT-4.

Same for inference speed/cost: many many incremental improvements within 1 year add up.